Let us put this concept to the test.
=> Read all tutorials in our Acceptance Testing series.
We are aware of what testing is. Acceptance denotes agreement or approval. A software product’s user can be the person who purchased it or the person who asked for it to be developed for them (client).
Thus, in accordance with my rule, the definition will be:
User Acceptance Testing (UAT), sometimes referred to as beta or end-user testing, is the process of having a client or user test software to see if it can be accepted.After functional, system, and regression testing are finished, this is the last test carried out.
The main purpose of this testing is to validate the software against the business requirements. The end-users who are familiar with the business requirements carried this validation out.
UAT, alpha, and beta testing are different types of acceptance testing.
As the user acceptance test is the last testing that is carried out before the software goes live, obviously this is the last chance for the customer to test the software and measure if it is fit for the purpose.
When Is It Performed
This is typically the last step before the product goes live or before the delivery of the product is accepted. This is performed after the product itself is thoroughly tested (i.e. after system testing).
Who Performs UAT
Users or clients – This could be either someone who is buying a product (with commercial software) or someone who has had software custom-built through a software service provider or the end-user if the software is made available to them ahead of time and when their feedback is sought out.
The team can include beta testers or the customer should internally choose UAT members from each organizational group to ensure testing of all user roles.
Need For User Acceptance Testing
Developers and functional testers are technical people who validate the software against the functional specifications. They interpret the requirements according to their knowledge and develop/test the software (here is the importance of domain knowledge).
This software is complete according to the functional specifications but there are some business requirements and processes that are known only to the end-users and are either communicated or misinterpreted.
This testing is crucial for validating whether or not all the business requirements are fulfilled before releasing the software for market use. Using live data and real use cases makes this testing an important part of the release cycle.
Many businesses that suffered big losses due to post-release issues know the importance of a successful User Acceptance Test. The cost of fixing the defects after release is many times greater than fixing it before.
Is UAT really necessary?
After performing loads of system, integration, and regression testing, one would wonder about the necessity of this testing. Speaking, this is the most important phase of the project as this is the time at which the users who are going to use the system would validate the system for its fit to purpose.
UAT is a test phase that largely depends on the perspective of the end-users and the domain knowledge of a department that represents the end-users.
It would be helpful to the business teams if they were involved in the project quite early so that they can provide their views and contributions that would help effective usage of the system in the real world.
User Acceptance Testing Process
The easiest way to understand this process is to think of this as an autonomous testing project – which means it will have the plan, design, and execution phases.
The following are the pre-requisites before the planning phase begins:
#1) Gather the key Acceptance Criteria
In simple terms, Acceptance criteria are a list of things that are going to be evaluated before accepting the product.
These could be of 2 types:
- Application Functionality or Business Related: Ideally, all the key business functionality should get validated, but due to various reasons, including time, it is not practical to do it all. Thus, meeting with the client or users involved in the testing can provide insights into the extent and focus of the testing.
- Contractual: We will not go into this and the involvement of the QA team in all this is almost nothing. The initial contract that gets drawn up even before the SDLC begins is reviewed and an agreement is reached upon whether all the aspects of the contract have been delivered or not.
We are going to focus only on the application functionality.
#2) Define the scope of QA involvement.
The QA team’s role is one of the following:
- No Involvement: This is very rare.
- Assist in this testing: Most common. Here, our involvement could train the UAT users on how to use the application and be on standby during this testing to make sure that we can help the users in case of any difficulty. Sometimes, besides being on standby and assisting, we might share their responses and record the results or log bugs, etc., while the users perform the actual testing.
- Perform UAT and present Results: If this is the case, the users will point to the areas of the AUT that they want to evaluate and the evaluation itself is performed by the QA team. Once completed, the QA team presents the results to the clients/users, who then decide if the results meet their expectations and if they are satisfied enough to accept the AUT. The decision is never that of the QA team.
Depending on the case on hand, we decide which approach is best.
The primary objectives and expectations:
Usually, UAT is undertaken by a Subject Matter Expert (SME) and /or a business user, who might be the owner or the customer of a system under test. Similar to the System testing phase, the UAT phase also encompasses religious phases before it is brought to closure.
Key activities of each UAT phase are defined below:
UAT Governance
Similar to system testing, the team enforces effective governance for UAT to ensure strong quality gates along with the defined Entry and Exit criteria (provided below **).
** Remember that this is only a form of guidance. The project’s needs and requirements may dictate modifications.
UAT Test Planning
The process is almost the same as with the regular test plan in the system phase.
The most common approach followed in most of the projects is to plan for both system and UAT testing phases together. For more information on the UAT test plan along with a sample, please check out the attached test plan document’s UAT sections.
User Acceptance Test Plan
(This is the same that you would find on our site for the QA training series as well).
Click on the below image and scroll down to find the test plan document sample in various formats. In that template, check the UAT section.
The dates, environment, actors(who), communication protocols, roles and responsibilities, templates, results, and their analysis process, entry-exit criteria–all of this and anything else that is relevant will be found in the UAT test plan.
Regardless of the QA team’s level of participation, we need to plan this phase thoroughly and consider all aspects.
=> Here is a User Acceptance Test Plan Sample Document
User Acceptance Testing Design
The gathered acceptance criteria from the users are used in this step. Samples could look as shown below.
(These are excerpts from CSTE CBOK. This is one of the best references available about this testing.)
User Acceptance Testing Template:
Based on the criteria, we (the QA team) give the users a list of UAT test cases. These test cases are not different from our regular system test cases. They are just a subset as we test all the applications as opposed, just to the key functional areas.
In addition to these, the data, templates to record test results, administrative procedures, defect logging mechanisms, etc., have to be in place before we move to the next phase.
Test Execution
Whenever feasible, this testing takes place in a conference or war room-style setting when members from the QA team, PM, and users gather for a day or two to go over all of the acceptance test cases.
We run the test cases on the AUT if the QA team is doing the tests.
The Acceptance Decision is made following the completion of all tests and receipt of the results. The Go/No-Go decision is another name for this. It’s a go if the users are happy, or a no-go otherwise.
Reaching the acceptance decision is typically the end of this phase.
Tools & Methodologies
Typically, the type of software tools that are used during this testing phase is similar to the tools used while performing functional testing.
Tools:
As this phase involves validating the complete end-to-end flows of the application, it might be difficult to have one tool to automate this validation completely. However, to some extent, we will leverage the automated scripts developed during system testing.
Similar to system testing, users would also use test management and defect management tools like QC, JIRA, etc. These tools can be configured to cumulate data for the User Acceptance phase.
Methodologies:
Though conventional methodologies such as specific business users performing UAT of the product are still relevant, in a truly global world like today, User Acceptance Testing sometimes has to involve varied customers across countries based on the product.
For Example, customers would use an e-commerce website across the globe. In scenarios like this, crowd-testing would be the best viable option.
Crowd testing is a methodology where people from all over the world can participate and validate the usage of the product and give suggestions and recommendations.
Crowd-testing platforms are built and are being used by many organizations now. The platform hosts a website or product that requires crowd-testing, and customers can nominate themselves to perform the validation. The feedbacks provided are then analyzed and prioritized.
Crowd Testing methodology is proving to be more effective as the pulse of the customer across the globe can be easily understood.
UAT In Agile Environment
The agile environment is more dynamic. In an agile world, business users will actively participate throughout the project sprints and the project team will enhance it based on their feedback loops.
At the beginning of the project, business users would be the key stakeholders in providing requirements and updating the product backlog. During the end of each sprint, business users would participate in the sprint demo and would be available to provide any feedback.
The team would plan a UAT phase before completing the sprint, where the business users would validate their work.
The feedbacks which are received during sprint demo and sprint UAT, are collated and added back to the product backlog, which is constantly reviewed and prioritized. Thus, in an agile world, the business users are closer to the project and they evaluate the same for its use more frequently, unlike the traditional waterfall projects.
UAT Team – Roles & Responsibilities
A Typical UAT organization would have the following roles and responsibilities. The project manager would support the UAT team, and development and testing teams based on their needs.
| Roles | Responsibilities | Deliverables |
|---|---|---|
| Business Programme Manager | • Create and maintain Programme Delivery plan • Review and Approve UAT Test Strategy and Plan • Ensure the successful completion of the programme on schedule and budget • Liaise with IT programme Manager and monitor the progress of the programme • Work closely with business operations team and equip them for Day 1 operation • Sign-off Business Requirement Document • Review the e-learning course content | • Programme progress report • Weekly status report |
| UAT Test Manager | • Crete UAT Strategy • Ensure effective collaboration between IT and Business BA and PMO • Participate in requirements walkthrough meetings • Review Effort Estimation, Test Plan • Ensure Requirement Traceability • Drive metrics collection to quantify the benefits derived out of the updated testing methodology, tools and environment usage | • Master Test Strategy • Review & approve Test Scenarios • Review & approve Test Cases • Review & Approve Requirement Traceability Matrix • Weekly Status report |
| UAT Test Lead & Team | • Verify & Validate Business Requirement against Business process • Estimation for UAT • Create & Execute UAT test Plan • Participate in requirement JAD session • Prepare test scenarios, test cases and test data based on Business Process • Maintain Traceability • Execute test cases and prepare test logs • Report defects in test management tool and manage them throughout their lifecycle • Produce UAT End of test report • Provide Business Readiness Support and Live proving | • Test Log • Weekly Status Report • Defect Report • Test Execution Metrics • Test Summary Report • Archived Reusable Test artifacts |
7 Challenges Of UAT And Mitigation Plan
It doesn’t matter if you are a part of a billion-dollar release or a startup team, overcome all these challenges to deliver successful software for the end-user.
#1) Environment setup and deployment process
Carrying out this test in the same environment used by the functional test team will certainly end up overlooking the real-world use cases. Also, crucial testing activities like performance testing can’t be carried out in an environment with incomplete test data.
A separate production-like environment should be set up for this test.
Once the UAT environment is separated from the test environment, you must control the release cycle effectively. An uncontrolled release cycle may lead to different software versions on the test and the UAT environment. If you do not test the software on the latest version, it wastes valuable acceptance test time.
Meanwhile, the time required for issue tracking on incorrect software versions is high.
#2) Test Planning
During the requirement analysis and design stage, this testing should be organized using a precise acceptance test strategy.
The set of practical use cases should be determined for implementation during strategy formulation. Determining the test objectives is crucial because huge applications cannot undergo a full test execution at this testing phase. The most important business goals should be prioritized before testing.
At the conclusion of the testing cycle, this testing is conducted. For the software release, this is the most crucial time. The UAT time will be consumed if any of the earlier phases of development and testing are delayed.
Improper test planning, in the worst cases, leads to an overlap between the system testing and UAT. The team deploys the software to this environment even if functional testing is not completed due to less time and pressure to meet deadlines. The core goals of this testing can’t be achieved in such situations.
The team should receive and review the UAT test plan well in advance of starting the test. This will assist them with test planning, test case writing, test script creation, and UAT environment setup.
#3) Handling new business requirements as incidents/defects
Ambiguities in requirements get caught in the UAT phase. UAT testers find issues arising due to ambiguous requirements (by looking at the complete UI which wasn’t available during the requirement gathering phase) and log it as a defect.
The customer expects these to be fixed in the current release without considering the time for the change requests. If a timely decision is not taken by the project management on these last-minute changes, then this could lead to the release failure.
#4) Unskilled testers or testers without business knowledge
When there is no permanent team, the company selects UAT staff from various internal departments.
Even if the staff is well familiar with the business needs, or if they lack training for the new requirements being developed, they cannot effectively perform UAT. Also, a non-technical business team might face many technical difficulties in executing the test cases.
Meanwhile, assigning testers at the end of the UAT cycle does not add any value to the project. Little time to train the UAT staff can significantly increase the chances of UAT success.
#5) Improper Communication Channel
Communication between remote development, testing, and the UAT team is more difficult. Email communication is often very difficult when you have an offshore tech team. A small ambiguity in incident reports can delay its fix for a day.
Proper planning and effective communication are critical to effective team collaboration. Project teams should use a web-based tool to log defects and questions. This will help to distribute the workload evenly and avoid reporting duplicate issues.
#6) Asking the Functional test team to perform this testing
Asking the functional test team to conduct UAT is the worst scenario imaginable.
Due to a shortage of resources, customers transfer their responsibilities to the test team. In these situations, the testing’s entire goal is jeopardized. End users will be able to identify problems that functional testers do not consider real-world situations as soon as the software is online.
Assigning this testing to committed, knowledgeable testers with business experience is one way to address this issue.
#7) The Blame Game
Business users occasionally merely look for excuses to reject the product. To win the business team’s respect, they could act superior by placing the blame on the development and testing team. Although it doesn’t happen often, teams might be impacted by internal politics.
It’s really challenging to handle such circumstances. Nonetheless, avoiding the blame game would be facilitated by cultivating a good rapport with the business team.
By overcoming these obstacles, I trust these suggestions will undoubtedly assist you in carrying out a successful user acceptance plan. The secret to successful user acceptability testing is a motivated team, effective planning, communication, and execution.
System Testing Vs User Acceptance Testing
The testing team is involved from the very beginning of the project, beginning with the requirement analysis stage.
Some sort of project validation is carried out at every stage of the project life cycle, such as regression, end-to-end, integration, unit, system, and static testing. This helps us better understand the testing that was done during the UAT phase and how it differs from the other testing that was done previously.
Even though SIT and UAT differ from one another, it’s crucial to take advantage of synergies while preserving the independence of each phase to allow for a quicker time to market.
Conclusion
The pages, forms, and buttons are not the focus of UAT. Even before this test starts, it is assumed that all of the fundamental components have been tested and are functioning well. If people found such a simple fault, it would be disastrous; the QA team would be horrified.
The entity that is the main component of the business is the subject of this testing.
Let me illustrate: The UAT will not be about looking for the menu that opens a page, etc., if the AUT is a ticketing system. It concerns the tickets and their reservation, the possible states, the system’s path, etc.
For instance, if the website is a car dealership, the emphasis is on the “car and its sales” rather than the website. The proprietors of the company are therefore the best people to verify and authenticate the main business. Therefore, when the client is heavily involved, this testing makes the most sense.
Since UAT is fundamentally testing, there’s a significant likelihood that some defects will be found during this stage as well. It does occur occasionally. In addition to being a significant escalation for the QA team, UAT defects typically require a meeting to sit and deliberate how to address them because, after testing, there is typically no time to resolve them and retest.
The decision would be either to:
- Push the go-live date, fix the issue first, and then move on.
- Leave the bug as it is.
- Consider it as a part of the change request for future releases.
UAT is classified as Alpha and Beta testing, but that classification is not that important in the context of typical software development projects in a service-based industry.
- Alpha testing is when UAT is carried out in the software builder’s environment and is more significant in the context of commercial off-the-shelf software.
- Beta testing is when the UAT is carried out in the production environment or the client’s environment. This is more common for customer-facing applications. The users here are the actual customers like you and me in this context.
Most of the time in a regular software development project, UAT is carried out in the QA environment if there is no staging or UAT environment.
In short, the best way to find out if your product is acceptable and fit for purpose is to actually put it in front of the users.
Organizations are getting into the Agile way of delivering, business users are getting more involved and the projects are being enhanced and delivered via feedback loops. All being done, the User Acceptance phase is considered the gate for getting into implementation and production.
YOU MAY BE INTERESTED IN
The Art of Software Testing: Beyond the Basics
Automation testing course in Pune
