This post covers an approach to handling User-Acceptance, Alpha, and Beta Testing in an Agile environment. In modern software development, there are many types of testing and it’s easy to confuse the purpose and timing of each. This is especially true for testing that happens outside of the core Development and Quality Assurance activities (such as unit, integration, and regression testing).
But UAT, Alpha, and Beta Testing are special. These forms of testing progressively bring in less controlled people and environments. Ultimately, you get invaluable insights into your product that benefit your Product Management, Development, Quality Assurance, UI/UX, Support, Sales, and Marketing teams.
We should also note that there are many variations of Agile and software development. Therefore, there are many variations of the concepts described below. This post provides a basic framework upon which to build a successful testing program, but do expect differences across different companies.
User Acceptance Testing (UAT)
The person asking for a software feature should be able to test that feature to ensure what was delivered matches their idea.
The Product Owner is responsible for maximizing product value and is accountable to all stakeholders, include internal and external users.
The Product Backlog is composed of User Stories. User Stories describe the user, the feature he/she wants to use and finally the reason he/she needs it.
- As a (user),
- I want (a feature)
- So I can (reason of need/business benefit)
User Stories should be small, negotiable, valuable, and testable.
A Sprint is a development cycle that focuses on a few requirements or User Stories.
Based on the above, the Product Owner has the primary responsibility for creating, prioritizing, and validating User Stories.
Why is UAT needed?
UAT is the step that validates the User Story. It ensures the Product Owner’s expectations are met.
When does UAT happen?
UAT is ongoing. In some environments, Sprints are delivered on a bi-weekly basis. This allows the Product Owner and the Development Team to work in an environment of continuous feedback.
User Stories do not need to be complete. Increments can and should be reviewed on a regular basis to allow for continuous feedback.
Who is involved in UAT?
- Product Owner
- Development Team (members who worked on the User Story)
- For highly-specialized features from other stakeholders (Finance, Marketing, Support), the Product Owner may involve member of the project team in UAT.
How is UAT performed?
When User Stories are created, Acceptance Criteria is established. Acceptance Criteria may evolve over time. Ultimately, the User’s objective for the User Story must be met.
How Does DevOps Relate to Agile?
Alpha Testing
As User Stories are put together for a release, that release should be tested outside of the Business Owner and Development Team. This will expose the software to people and environments outside of the project team.
The people and environments tend to act in a less predictable manner, often uncovering issues that are not found in a controlled environment.
Alpha testers should expect bugs, performance issues, crashes, and little-to-no documentation.
Why is Alpha Testing needed?
Alpha Testing helps ensure the software is behaves as expected across a broader set of users and environments. Alpha Testing should identify show-stoppers that will prevent Beta Testing.
When does Alpha Testing happen?
Alpha occurs after all features are developed, have been tested through the Development/QA organizations, and have been signed off by the Product Owner (via UAT).
Alpha Testing phases typically run 1-2 weeks, but this varies widely based on the size and complexity of the release.
A Development phase may be planned that overlaps as follows:
- Alpha weeks 1-2
- Development weeks 2-3
PRO TIP:
“Treat bugs in new features and regressions in existing features differently. If a bug surfaces during development, take the time to understand the mistake, fix it, and move on. If a regression appears (i.e., something worked before but doesn’t anymore), then it’s likely to reappear. Create an automated test to protect against that regression in the future.” Atlassian Agile Coach
Who participates in Alpha Testing?
- Testers: Testers are typically internal to the company but not part of the Project Team.
- Supporters: Product Owner, Development, Quality Assurance, UI/UX, and Support teams to support users, analyze feedback in real time, and document as appropriate.
How is Alpha Testing performed?
Establish a plan that includes:
- Who is testing
- Who is supporting
- What they’ll be testing (test cases covering all features)
- When they’ll be testing
- How issues will be managed (reporting, logging, classification, and prioritization)
- Exit criteria (all test cases run, no “show stopper” defects)
Defect handling:
- Defects are logged and evaluated in real time.
- Some defects may be fixed in real time; however, most defects require deeper evaluation and may trigger the need for more complete development, integration, and regression testing.
- Some defects will be placed for future development; others may be trigger a “no go” decision.
Beta Testing
Once a software release has passed Alpha Testing, it should be exposed outside of the organization to people within the target market. This step provides feedback from actual customers on satisfaction in terms of the user interface, features, functions, and overall experience with the software.
Beta testers should expect software that has bugs, crashes, and incomplete documentation.
Why is Beta Testing needed?
Beta provides feedback from real customers in actual customer environments.
When does Beta Testing happen?
Beta occurs after Alpha Testing and prior to the Release Candidate build.
Beta Testing phases typically run 1-2 weeks, but this varies widely based on the size and complexity of the release.
A Development phase may be planned that overlaps as follows:
- Beta weeks 1-2
- Development weeks 2-3
In some cases, Beta can be an ongoing activity. Some customers may voluntarily choose to be part of a “beta channel” whereby they are receiving updates to existing software ahead of production releases. In this case, sophisticated tracking and logging mechanisms are built into the software so that the Supporters can gain a full set of insights and adjust the software before it hits the standard channel.
As an example, Google allows you to select from different channels on Chrome OS, which include Stable, Beta, and Dev.
Who participates in Beta Testing?
- Testers: Beta testers are typically actual end-customers (from the target market) who have received the software for free or for incentive.
- Supporters: Product Owner, Development, Quality Assurance, UI/UX, and Support teams to support users, analyze feedback in real time, and document as appropriate.
How is Beta Testing performed?
Establish a plan that includes:
- Who is testing
- Who is supporting
- What they’ll be testing (test cases covering all features)
- When they’ll be testing
- How issues will be managed (reporting, logging, classification, and prioritization)
- Exit criteria (all test cases run, no “show stopper” defects)
Defect handling:
- Defects are logged and evaluated in real time.
- Some defects may be fixed in real time; however, most defects require deeper evaluation and may trigger the need for more complete development, integration, and regression testing.
- Some defects will be placed for future development; others may be trigger a “no go” decision.
How Does Scrum Work for Geographically Distributed Teams?
Conclusion
Managing and performing these types of tests is a key part of successful software delivery. Whether you follow this plan, something more basic, or something more sophisticated, the goal is to gain continuous feedback and as a result, deliver a better product.
Tags:
