Acceptance Testing

For your better comprehension, this special acceptance testing lesson will provide you with a thorough rundown of the types, applications, meaning, and other aspects of acceptance tests in an easy-to-understand format.

What is Acceptance Testing?

Once the System Testing process is completed by the testing team and is signed-off, the entire Product/application is handed over to the customer/few users of customers/both, to test for its acceptability i.e., Product/application should be flawless in meeting both the critical and major Business requirements. Also, end-to-end business flows are verified similar to in real-time scenarios.

The production-like environment will be the testing environment for Acceptance Testing (Usually termed as Staging, Pre-Prod, Fail-Over, UAT environment).

This is a black-box testing technique where only the functionality is verified to ensure that the product meets the specified acceptance criteria (no need for design/implementation knowledge).

Why Acceptance Tests?

Though System testing has been completed successfully, the Acceptance test is demanded by the customer. Tests conducted here are repetitive, as they would have been covered in System testing.

So why is this testing being conducted by customers?

This is because:

• To gain confidence in the product that is getting released to the market.
• This is to ensure that the product is working the way it has to.
• This is to ensure that the product matches current market standards and is competitive enough with other similar products in the market.

Types

There are several types of this testing.

A few of them are listed below:

#1) User Acceptance Testing (UAT)

The purpose of UAT is to determine whether the product is functioning properly for the user. In order to do testing, specific needs that are often utilized by end users are chosen. Another name for this is end-user testing.

In this context, the term “User” refers to the end users to whom the product or program is meant; as a result, testing is done from their viewpoint.

=> Also Read: What is User Acceptance Testing (UAT)?

#2) Business Acceptance Testing (BAT)

This is to assess whether the Product meets the business goals and purposes or not.

BAT mainly focuses on business benefits (finances) which are quite challenging due to the changing market conditions/advancing technologies so the current implementation may have to undergo changes that result in extra budgets.

Even the Product passing the technical requirements may fail BAT due to these reasons.

#3) Contract Acceptance Testing (CAT)

This contract stipulates that the product must pass all acceptance use cases and undergo an acceptance test within a specified time frame after becoming live.

The provisions of this contract, which is known as a Service Level Agreement (SLA), provide that payment will only be given if the product services meet all requirements, indicating that the contract has been satisfied.

This contract may occasionally be made before to the product’s launch. In any case, a contract should be clearly specified regarding the testing period, the testing locations, the conditions for problems that may arise later, payments, etc..

#4) Regulations/Compliance Acceptance Testing (RAT)

This is to determine whether the product is in violation of the laws and guidelines established by the national government of the country in which it is being distributed. Even though it might not have been intended, this will hurt the company.

A produced product or application that is meant to be launched globally typically needs to go through RAT since different nations and regions have distinct laws and regulations that are established by their own governing bodies.

A country or a particular area within a country will not be permitted to utilize the product and will be deemed to have failed if any of the rules and regulations are broken. Even if there is a breach, the product’s vendors will bear direct responsibility if the product is published.

#5) Operational Acceptance Testing (OAT)

This is to assess the operational readiness of the Product and is non-functional testing. It mainly includes testing of recovery, compatibility, maintainability, technical support availability, reliability, fail-over, localization, etc.

OAT mainly assures the stability of the product before releasing it to production.

#6) Alpha Testing

This is to assess the Product in the development/testing environment by a specialized tester team usually called alpha testers. Here’s the tester’s feedback and suggestions to help improve Product usage and also to fix certain bugs.

Here, testing happens in a controlled manner.

=> Also Read: What is Alpha Testing?

#7) Beta Testing/Field Testing

This is to assess the Product by exposing it to real end-users, usually called beta testers/beta users, in their environment. Continuous feedback from the users is collected and the issues are fixed. Also, this helps in enhancing/improving the Product to give a rich user experience.

Testing happens in an uncontrolled manner, which means a user has no restrictions on the way in which the Product is being used.

=> Also Read: What is Beta Testing?

All these types have a common goal:

• Ensure to gain/enrich Confidence in the Product.
• Ensure that the Product is ready to be used by real users.

Who does Acceptance Testing?

For the Alpha type, only the members of the organization (who developed the Product) perform the testing. These members are not directly a part of the project (Project managers/leads, developers, testers). Management, Sales, and Support teams usually perform the testing and provide feedback accordingly.

Apart from the Alpha type, all other acceptance types are generally performed by different stakeholders. Like customers, customer’s customers, specialized testers from the organization (not always).

It is also good to involve Business Analysts and Subject Matter Expertise while performing this testing based on its type.

Qualities of Acceptance Testers

Testers with the below qualities are qualified as Acceptance testers:

• Ability to think logically and analytically.
• Good domain knowledge.
• Able to study the competitive products in the market and analyze the same in the developed product.
• Having an end-user perception while testing.
• Understand the business needs for each requirement and test accordingly.

Impact of Issues found during this testing

Any issues encountered in the Acceptance test phase should be considered a high priority and fixed immediately. This also requires Root Cause Analysis to be performed on each and every issue that is found.

The testing team plays a major role in providing RCA’s for Acceptance issues. This also helps in determining how efficient testing is performed.

Also, valid issues in the acceptance test will hit both the testing and development team efforts in terms of impressions, ratings, customer surveys, etc. Sometimes, if any ignorance from the testing team on validations is found, it leads to escalations as well.

Use

This testing is useful in several aspects.

A few of these include:

• Figure out the issues missed during the functional testing phase.
• How well the product is developed.
• A product is what the customers need.
• Feedback/surveys conducted help in improving product performance and user experience.
• Improve the process followed by having RCAs as input.
• Minimise or eliminate issues arising from the Production Product.

Differences between System Testing, Acceptance Testing, and User Acceptance Testing

Given below are the prime differences between these 3 types of Acceptance tests.

System Testing	Acceptance Testing	User Acceptance Testing
End-to-end testing is performed to verify whether Product meets all the specified requirements	Testing is performed to verify whether Product meets customer requirements for acceptability	Testing is performed to verify whether end-users requirements are fulfilled for acceptability
A product is tested as the whole focusing only on functional and non-functional needs	Product is tested for business needs – user acceptability, business goals, rules and regulations, operations, etc.	Product is tested only for user acceptability
Testing team performs System Testing	Customer, Customers’ customers, tester (rarely), management, Sales, Support teams performs acceptance testing depending on the type of test carried out	Customer, Customers’ customer, testers (rarely) performs user acceptance testing
Test cases are written and executed	Acceptance tests are written and executed	User Acceptance tests are written and executed
Can be functional and non-functional	Usually Functional, but non-functional in case of RAT, OAT, etc	Only Functional
Only test data is used for testing	Real-time data/production data is used for testing	Real-time data / Production data is used for testing
Positive and negative tests are performed	Usually Positive tests are performed	Only Positive tests are performed
Issues found are considered as bugs and fixed based on severity and priority	Issues found marks Product as Failure, and considered to be fixed immediately	Issues found marks Product as Failure and considered to be fixed immediately
Controlled manner of testing	Can be controlled or uncontrolled based on type of testing	Uncontrolled manner of testing
Testing on Development environment	Testing on Development environment or pre-production environment or production environment, based on type	Testing is always on Pre-Production environment
No assumptions, but if any can be communicated	No assumptions	No assumptions

Acceptance Tests

Similar to Product test cases, we do have acceptance tests. Acceptance tests are derived from User Stories’ acceptance criteria. These are usually the scenarios that are written at a high-level detailing what the Product has to do under different conditions.

It does not give a clear picture of how to perform tests, as in test cases. Acceptance tests are written by Testers who have a complete grip on the Product, usually Subject Matter Expertise. All the tests written are reviewed by the customer and/or business analysts.

These tests are executed during the acceptance test. Along with acceptance tests, a detailed document on any set-ups to be done has to be prepared. It should include every minute details with proper screenshots, set-up values, conditions, etc.

Acceptance Test Bed

Although it is a different testbed, the testbed used for this testing is comparable to a standard one. Platforms must be configured very similarly to the production environment, including the necessary hardware, software, operating products, network, server, database, licensing, plug-ins, and other components.

The platform or environment where the planned acceptance tests will be carried out is known as the acceptance testbed. It is a good idea to check for environmental problems and product stability before delivering the acceptance test environment to the customer.

A standard testing environment can be utilized for acceptance testing if no special environment has been built up for it. It will be disorganized here, though, because the test data from routine system testing and the real-time data from acceptance testing are maintained in a single environment.

The acceptance testbed is usually set up on the customer-side (i.e., in the laboratory) and will have restricted access to the development and testing teams.

Teams will be required to access this environment through VMs/or specifically designed URLs using special access credentials, and all access to this will be tracked. Nothing in this environment has to be added/modified/deleted without the customer’s permission, and they should be notified of the changes that are made.

Entry and Exit Criteria for AT

The Acceptance Test Plan, which is discussed in the second section of this tutorial, should clearly specify the entry and exit criteria for acceptance testing, just like it does for any other phase of the STLC.

This stage begins immediately following system testing and concludes prior to the production launch. Consequently, the system testing exit criteria have been incorporated into the AT entry requirements. In a similar vein, the AT exit criteria are now included in the Production Launch entry requirements.

Entry Criteria

Given below are the conditions to be fulfilled before starting:

• Business requirements should be clear and available.
• The system and Regression testing phase should be completed.
• All the Critical, Major & Normal bugs should be fixed and closed (Minor bugs accepted mainly are cosmetic bugs that do not disturb the usage of the product).
• Known issues list should be prepared and shared with the stakeholders.
• Acceptance Test Beds should be set up and a high-level check should be performed for no environmental issues.
• The System Testing phase should be signed-off letting the product move to the AT phase (Usually done through email communication).

Exit Criteria

There are certain conditions that need to be fulfilled by AT to allow the product to go for a Production Launch.

They are as follows:

• Acceptance tests should be executed and all the tests should pass.
• No Critical/Major defects left Open. All defects should be fixed and verified immediately.
• AT should be Signed-off-by all the included stakeholders with a Go/No-Go Decision on the product.

Acceptance Testing Process

In V-Model, the AT phase is in parallel to the Requirements phase.

The actual AT process goes as shown below:

Business Requirements Analysis

Business requirements are analyzed by referring to all the available documents within the project.

Some of which are:

• System Requirement Specifications
• Business Requirements Document
• Use Cases
• Workflow diagrams
• Designed data matrix

Design Acceptance Test Plan

There are certain items that need to be documented in the Acceptance Test Plan.

Let’s take a look at some of them:

• Acceptance Testing strategy and approach.
• Entry and exit criteria should be well-defined.
• The scope of AT should be well-mentioned and it has to cover only the business requirements.
• The acceptance test design approach should be detailed so that anyone writing tests can easily understand the way in which it has to be written.
• Test Bed set up and actual testing schedule/timelines should be mentioned.
• As testing is conducted by different stakeholders, details on logging bugs should be mentioned as the stakeholders may not be aware of the procedure followed.

Design and Review Acceptance Tests

Acceptance tests should be written at a scenario level mentioning what has to be done (not in detail to include how to do it). These should be written only for the identified areas of scope for business requirements, and each and every test has to be mapped to its referencing requirement.

All written acceptance tests have to be reviewed to achieve high coverage of business requirements.

This is to ensure that all other tests apart from the scope mentioned are not involved so that testing lies within the scheduled timelines.

Acceptance Test Bed Set up

Test Beds should be set up similar to a Production environment. Very high-level checks are required to confirm environment stability and usage. Share your credentials to use the environment only with the stakeholder who is performing this testing.

Acceptance Test Data Set-Up

Production data has to be prepared/populated as test data in the systems. Also, there should be a detailed document in such a way that the data has to be used for testing.

Do not have the test data like TestName1, TestCity1, etc., instead I have Albert, Mexico, etc. This gives a rich real-time data experience and testing will be up-to-the-point.

Acceptance Test Execution

Designed Acceptance tests have to be executed in the environment at this step. Ideally, all the tests should pass at the first attempt itself. There should be no functional bugs arising out of Acceptance testing, if any, then they should be reported as a high priority to be fixed.

Again, bug fixes have to be verified and closed as a high-priority task. Test execution reports have to be shared on a daily basis.

Bugs logged in this phase should be discussed in a bug-triage meeting and have to undergo the Root Cause Analysis procedure. This is the only point where acceptance testing assesses whether all the business requirements are met by the product or not.

Business Decision

A go/no-go decision has come out for the product to be launched in Production. Go decision will take the product ahead to be released to the market. A no-go decision marks the product as a failure.

A few factors related to the No-Go Decision:

• Poor quality of the product.
• Too Many open Functional Bugs.
• Deviation from business requirements.
• This is not up to market standards and needs enhancements to match current market standards.

Success factors for This Testing

Once this test is planned, prepare a checklist that increases its success rate. There are some action items that are to be followed before the Acceptance test starts.

They are:

• Have a well-defined scope and make sure there is a business need for the scope identified for this testing.
• Execute Acceptance tests in the System testing phase itself at least once.
• Perform extensive ad-hoc testing for each of the acceptance test scenarios.

Conclusion

In a nutshell, Acceptance testing helps in figuring out the efficiency of development and testing teams.

There are several tools to conduct this activity, but usually, it is preferred to be done manually as there is an involvement of the real users and different stakeholders who are not from a technical background, and it may not be feasible for them.

YOU MAY BE INTERESTED IN

The Art of Software Testing: Beyond the Basics

Automation testing course in Pune

Automation testing in selenium

Mastering Software Testing: A Comprehensive Syllabus