Regression Testing is done to ensure that the product works fine with new functionality, bug fixes, or any changes to the existing feature. Previously executed test cases are re-executed to verify the impact of the change.
Watch this video on Regression Testing in Software Testing:
Regression testing can be performed on a new build when there is a significant change in the original functionality that too even in a single bug fix. Regression means retesting the unchanged parts of the application.
Tutorials Covered in This Series:
Tutorial #1: What is Regression Testing (This Tutorial)
Tutorial #2: Regression Test Tools
Tutorial #3: Retest Vs Regression Testing
Tutorial #4: Automated Regression Testing in Agile
Regression testing is like a verification method. Test cases are generally automated as test cases are required to be executed again and running the same test cases again manually is time-consuming and tedious too.
For Example, Consider a product X, in which one of the functionality is to trigger confirmation, acceptance, and dispatched emails when the Confirm, Accept and Dispatch buttons are clicked.
The confirmation email had some problems, therefore some code changes were made to address these. It is necessary to test not only the confirmation emails but also the acceptance and dispatched emails to make sure the code update hasn’t impacted them.
Java, C++, C#, and other programming languages are not necessary for regression testing. This testing technique is used to check for changes or updates being made to the product. It confirms that any changes made to a product don’t impact its current modules.
Make that the bug has been fixed and that the recently introduced features haven’t caused any issues with the previous functional version of the software.
Testers perform Functional Testing when a new build is available for verification. The intent of this test is to verify the changes made in the existing functionality and the newly added functionality as well.
When this test is done, the tester should verify whether the existing functionality is working as expected and the new changes have not introduced any defect in functionality that was working before this change.
Regression testing should be part of the Release Cycle and must be considered in the test estimation.
Why is Regression Testing Important?
Regression is initiated when a programmer fixes any bug or adds a new code for new functionality to the system.
Both the recently created and pre-existing functionality may have numerous dependencies.
In order to ensure that the unaltered code is not impacted, this quality check verifies if the updated code agrees with the old code. The testing team is typically tasked with verifying any last-minute system modifications.
Only the application area needs to be tested in this scenario in order to finish the testing process on schedule and cover all of the important system components.
This test is crucial if the program is continuously improved or changed. The tested code should be unaffected by the new feature.
Regression is required to find the bugs that occurred because of a change in the code. If this testing is not done, the product might get critical issues in the live environment and that indeed can lead the customer into trouble.
While testing any online website, the tester reports an issue that the Price of the Product is not showing correctly i.e., it shows a lesser price than the actual price of the Product, and it needs to be fixed soon.
Once the developer fixes the issue, it needs to be re-tested and Regression Testing is also required as verifying the price on the reported page would have got corrected but it might be showing an incorrect price on the summary page where the total is shown along with the other charges or the mail sent to the customer still has the incorrect price.
Now, in this case, the customer will have to bear the loss if this testing is not performed as the site calculates the total cost with the incorrect price and the same price goes to a customer by email. Once the customer accepts, the Product is sold online at a lower price, it will be a loss for the customer.
This testing plays a big role and is very much required and important as well.
When to Perform Regression Tests?
After updates or new functionality have been verified, regression testing is typically carried out. However, this isn’t always true. Regression tests need to be included in the daily test cycle for releases that are taking months to finish. Regression testing can be done for weekly releases once the modifications have undergone functional testing.
Retesting, or just repeating a test, is a form of regression checking. The cause for retesting could be anything. Assume that you were testing a feature at the end of the day and were unable to complete the test, forcing you to conclude the process before determining whether the test was successful or unsuccessful.
The next day when you come back, you perform the test once more – that means you are repeating a test you performed before. The simple act of repeating a test is a Retest.
Regression testing at its core is a retest of sorts. It is only for the special occasion that something in the application/code has changed. It might be code, design or anything at all that dictates the overall framework of the system.
A retest that is conducted in this situation to make sure that the said change has not made an impact on anything that was already working before is called the Regression Test.
The most common reason why this might be conducted is because new versions of the code have been created (increase in scope/requirement) or bugs have been fixed.
Can Regression Testing be Performed Manually?
I was just teaching one of these days in my class, and a question came to me – “Can regression be done manually?”
I answered the question and we moved on in the class. Everything seemed OK, but somehow this question nagged me for quite a while later.
Over many batches, this question comes multiple times in various different ways.
Some of them are:
- Do we need a tool to perform the test execution?
- How is Regression Testing performed?
- Even after an entire round of testing– newcomers find it difficult to discern what exactly the Regression test is?
Of course, the original question:
- Can this Testing be performed manually?
To begin with, Test execution is a simple act of using your Test cases and performing those steps on the AUT, supplying the test data and comparing the result obtained on the AUT with the expected result mentioned in your test cases.
Depending on the comparison result, we set the status of the test case pass/fail. Test execution is as simple as that, there are no special tools necessary for this process.
Top Automated Regression Testing Tools
The majority of the testing may be automated via the Automated Regression Test. Using a fresh build, we ran every test case that had been run before.
This indicates that we have a set of test cases at our disposal, and that manually executing them takes a lot of time. Automating these test cases saves time and is an effective regression test technique because we are aware of the expected outcomes. The number of test cases that will continue to be relevant throughout time determines how much automation is necessary.
Automating the regression process will be a waste of time if test cases change periodically and the application’s scope continues to grow.
Most of the Regression testing tools are of record and playback types. You can record the test cases by navigating through the AUT (application under test) and verify whether the expected results are coming or not.
Recommended Tools:
#1) Avo Assure
Avo Assure is a 100% no-code and heterogeneous test automation solution that makes regression testing simpler and faster.
Its cross-platform compatibility enables you to test across the web, mobile, desktop, Mainframe, ERPs, associated emulators, and more. With Avo Assure, you can run end-to-end regression tests without writing a single line of code to ensure rapid, high-quality delivery.
Avo Assure can help you to:
- Achieve >90% test automation coverage by executing end-to-end regression tests repeatedly.
- Easily visualize your entire testing hierarchy with a click of a button. Define test plans and design test cases through the Mindmaps feature.
- Leverage about 1500+ keywords and >100 SAP-specific keywords to deliver applications faster
- Execute multiple scenarios simultaneously using the Smart Scheduling and Execution feature.
- Integrate with a plethora of SDLC and Continuous Integration solutions like Jira, Sauce Labs, ALM, TFS, Jenkins, and QTest.
- Analyze reports intuitively with easy-to-read screenshots and videos of test case execution.
- Enable accessibility testing for your applications.
#2) Katalon
Katalon is an all-in-one platform for test automation with a large user community. It offers free and codeless solutions to automate regression testing. Since it’s a ready-made framework, you can use it right away. No complicated setup is needed.
You can:
- Quickly create automated test steps using Record and Playback.
- Easily capture test objects and maintain them in a built-in repository (page-object model).
- Reuse test assets to scale up the number of automated regression tests.
It also provides more advanced features (like built-in keywords, scripting mode, self-healing, cross-browser testing, test reporting, CI/CD integration, and more) to help QA teams meet their extended testing needs when scaling up.
#3) DogQ
DogQ is a no-code automation testing tool and is suitable for both beginners and professionals. The tool is equipped with a bunch of cutting-edge features for creating various types of tests for websites and web apps, including regression testing.
The product allows users to run multiple test cases in the cloud and manage them directly through a custom-built interface. The tool uses AI-based text recognition technology that works for users automatically and provides them with 100% readable and editable test results. Moreover, test cases and scenarios can be run simultaneously, scheduled, edited, and then easily reviewed by non-technical team members.
DogQ is a perfect solution for startups and individual entrepreneurs who don’t have a lot of resources to test their websites and apps, or who don’t have the experience to do it themselves. DogQ offers flexible pricing plans starting from 5$ per month.
All pricing plans are based only on the number of steps a company may need for testing processes. Other advanced features such as integration, parallel testing, and scheduling are available with DogQ for use by all companies without the need to upgrade the plan.
#4) Virtuoso
Virtuoso delivers self-healing tests, eliminating the need to tinker with faulty tests in your regression pack with each release. Virtuoso starts bots that explore the DOM of the application and create a detailed model of every element using the selectors, IDs, and attributes that are accessible. Every test run employs a machine learning technique to detect any unexpected changes intelligently, allowing testers to focus on detecting flaws rather than repairing tests.
Similar to writing a manual test script, regression tests are written in plain English using Natural Language Programming. With the speed and accessibility of a codeless tool, this scripted technique yet has all the flexibility and power of a coded approach.
- Cross-browser and cross-device, write one test for everywhere.
- The fastest authoring experience.
- A next-generation AI-augmented testing tool.
- Guaranteed in-sprint regression testing.
- Out-of-the-box integration with your CI/CD pipeline.
#5) BugBug
BugBug is probably the simplest way to automate your regression testing. All you have to do is “record & replay” your tests with an intuitive interface.
How Does It Work?
- Create a test scenario
- Start recording
- Just click on your website – BugBug records all your interactions as test steps.
- Run your test – BugBug repeats all your recorded test steps.
A Simpler Alternative to Selenium
- Easier to learn
- Faster creation of production-ready regression tests.
- Doesn’t require coding
Good value for money:
- FREE if you only run automated regression tests in your local browser.
- For only $49 monthly you can use BugBug cloud to run all your regression tests every hour.
Further Reading => Detailed Review on BugBug Edit & Rewind Feature
Other Regression testing tools:
- Selenium
- AdventNet QEngine
- Regression Tester
- vTest
- Watir
- actiWate
- Rational Functional Tester
- SilkTest
Most of these are Functional and Regression test tools.
Recommended Reading => Check Here For The List Of Top Regression Tools
Adding and updating Regression test cases in an Automation test suite is a cumbersome task. While selecting an Automation tool for Regression tests, you should check if the tool allows you to add or update test cases easily.
In most cases, we need to update automated Regression test cases frequently due to frequent changes in the system.
Types of Regression Testing
Given below are the various types of Regression :
- Unit Regression
- Partial Regression
- Complete Regression
#1) Unit Regression
Unit Regression is done during the Unit Testing phase and code is tested in isolation i.e. any dependencies on the unit to be tested are blocked so that the unit can be tested individually without any discrepancy.
#2) Partial Regression
Partial Regression is done to verify that the code works fine even when the changes have been done in the code and that unit is integrated with the unchanged or already existing code.
#3) Complete Regression
Complete Regression is done when a change in the code is done on a number of modules and also if the change impact of a change in any other module is uncertain. The product as a whole is regressed to check for any changes because of the changed code.
How Much Regression Is Required?
This depends upon the scope of the newly added features.
The application area that is impacted is also fairly big if the scope of a fix or feature is too broad, so testing should be done carefully, taking into account all application test cases. However, when the tester receives feedback from a developer regarding the extent, type, and magnitude of change, this can be successfully determined.
Since these tests are repetitive, they may be automated, making it simple to run just a set of them on a fresh build.
In order to cover the most functionality in the fewest number of test cases, regression test scenarios must be carefully chosen. Continuous enhancements are required for this set of test cases in order to accommodate newly added features.
It becomes very difficult when the application scope is very huge and there are continuous increments or patches to the system. In such cases, selective tests need to be executed in order to save testing cost and time. These selective test cases are picked based on the enhancements done to the system and the parts where it can affect the most.
What Do We Do With Regression Checks?
- Re-run the previously conducted tests.
- Compare the current results with previously executed test results
This is a continuous process performed at various stages throughout the software testing lifecycle.
For a short release, it is best practice to do a regression test at the conclusion of functional testing and following sanity or smoke testing.
The creation of a regression test plan is necessary for efficient testing. The exit criterion and the regression testing approach should be described in this plan. This test also includes performance testing to ensure that modifications to the system components do not impair system performance.
Best practices: To ensure that any regression side effects are addressed in the following day’s build, run automated test cases every day in the evening. By addressing nearly all regression flaws early on rather than identifying and resolving those, it lowers the release risk at the end of the release cycle.
Regression Testing Techniques
Given below are the various techniques.
- Retest all
- Regression Test Selection
- Test case Prioritization
- Hybrid
#1) Retest All
As the name itself suggests, the entire test cases in the test suite are re-executed to ensure that there are no bugs that have occurred because of a change in the code. This is an expensive method as it requires more time and resources when compared to the other techniques.
#2) Regression Test Selection
In this method, test cases are selected from the test suite to be re-executed. Not that the entire suite has been re-executed. The selection of test cases is done on the basis of code change in the module.
Test cases are divided into two categories, one is Reusable test cases and another one is Obsolete test cases. The reusable test cases can be used in future regression cycles whereas obsolete ones are not used in the upcoming regression cycles.
#3) Test Case Prioritization
Test cases with high Priority are executed first rather than the ones with medium and low priority. The priority of the test case depends on its criticality and its impact on the product and also on the functionality of the product which is used more often.
#4) Hybrid
The hybrid technique is a combination of Regression Test Selection and Test case Prioritization. Rather than selecting the entire test suite, select only the test cases which are re-executed depending on their priority.
How To Select A Regression Test Suite?
Most of the bugs found in the production environment occur because of the changes done or bugs fixed at the eleventh hour i.e., the changes done at a later stage. The bug fix at the last stage might create other issues/bugs in the Product. That’s why Regression checking is very important before releasing a Product.
Below is a list of test cases that can be used while performing this Test:
- Functionalities that are frequently used.
- Test cases that cover the module where the changes have been made.
- Complex test cases.
- Integration test cases that include all the major components.
- Test cases for the core functionality or features of the Product.
- Priority 1 and Priority 2 test cases should be included.
- Test cases of frequently failed or recent testing defects were found for the same.
How To Perform Regression Testing?
Now that we have established what regression means, it is apparent that it is testing also – simply repeating in a specific situation for a specific reason. Therefore, we can safely derive that the same method applied for testing in the first place can be applied to this too.
Therefore, if testing can be done manually then Regression Testing can be done too. The use of a tool is not necessary. However, as time goes on applications get piled on with more and more functionality which keeps increasing the scope of regression. To make the most of the time, this testing is most often Automated.
Given below are the various steps involved in performing this Testing
- Prepare a Test suite for Regression considering the points mentioned in “How to select Regression Test suite”?
- Automate all test cases in the test suite.
- Update the Regression suite whenever it is required like if any new defect which is not covered in the test case is found, and a test case for the same should be updated in the test suite so that the testing is not missed for the same next time. The regression test suite should be managed properly by continuously updating the test cases.
- Execute the Regression test cases whenever there is any change in the code, the bug is fixed, new functionality is added, an enhancement to the existing functionality is done, etc.
- Create a test execution report which includes the Pass/Fails status of the executed test cases.
For Example:
Let me explain this with an example. Please examine the situation below:
| Release 1 Statistics | |
|---|---|
| Application Name | XYZ |
| Version/Release Number | 1 |
| No. of Requirements (Scope) | 10 |
| No. of Test Cases/Tests | 100 |
| No. of days it takes to Develop | 5 |
| No. of days it takes to Test | 5 |
| No. of Testers | 3 |
| Release 2 Statistics | |
|---|---|
| Application Name | XYZ |
| Version/Release Number | 2 |
| No. of Requirements (Scope) | 10+ 5 new Requirements |
| No. of Test cases/Tests | 100+ 50 new |
| No. of days it takes to Develop | 2.5 (since this half the amount of work than earlier) |
| No. of days it takes to Test | 5(for the existing 100 TCs) + 2.5 (for new Requirements) |
| No. of Testers | 3 |
| Release 3 Statistics | |
|---|---|
| Application Name | XYZ |
| Version/Release Number | 3 |
| No. of Requirements (Scope) | 10+ 5 + 5 new requirements |
| No. of Test cases/Tests | 100+ 50+ 50 new |
| No. of days it takes to Develop | 2.5 (since this half the amount of work than earlier) |
| No. of days it takes to Test | 7.5 (for the existing 150 TCs) + 2.5 (for new Requirements) |
| No. of Testers | 3 |
Given below are the observations that we can make from the above situation:
- As the releases grow, the functionality grows.
- Development time does not necessarily grow with releases, but the testing time does.
- No company/its management will be ready to invest more time in testing and less for development.
- We cannot even reduce the time it takes to test by increasing the test team size because more people means more money and new people also means lots of training and maybe also a compromise in quality as the new people might not be at par with the required knowledge levels immediately.
- The other alternative is clearly to reduce the amount of regression. But that could be risky for the software product.
For all these reasons, Regression Testing is a good candidate for Automation Testing, but it does not have to be done only that way.
Basic Steps to Perform Regression Tests
Every time the software undergoes a change and a new version/release comes up, given below are the steps that you can take to carry out this type of testing.
- Understand what kind of changes have been made to the software
- Analyze and determine what modules/parts of the software might be impacted – the development and BA teams can be instrumental in providing this information.
- Take a look at your test cases and determine if you will have to do a full, partial or unit regression. Identify the ones that will fit your situation
- Schedule a time and test away!
Regression in Agile
Agile is an adaptive approach that follows an iterative and incremental method. The product is developed in a short iteration called sprint which lasts for 2- 4 weeks. In agile, there are a number of iterations, hence this testing plays a significant role as the new functionality or code change is done in the iterations.
The Regression test suite should be prepared from the initial phase and should be updated with each sprint.
In Agile, Regression checks are covered under two categories:
- Sprint Level Regression
- End to End Regression
#1) Sprint Level Regression
Sprint Level Regression is done mainly for new functionality or enhancements that are done in the latest sprint. Test cases from the test suite are selected as per the newly added functionality or the enhancement that is done.
#2) End-to-End Regression
End-to-End Regression includes all the test cases that are to be re-executed to test the complete product end to end by covering all the core functionalities of the Product.
Agile has short sprints and as it goes on, it is very much required to automate the test suite, the test cases are executed again and that too needs to be completed in a short span of time. Automating the test cases reduces the time of execution and defect slippage.
Advantages
Given below are the various advantages of the Regression test
- It improves the quality of the Product.
- This ensures that any bug fixes or enhancements that are done do not impact the existing functionality of the Product.
- Automation tools can be used for this testing.
- This will ensure that issues that are already fixed do not occur again.
Disadvantages
Though there are several advantages, there are some disadvantages as well. They are:
- This has to be done for a small change in the code as well because even a small change in the code can create issues in the existing functionality.
- If in case automation is not used in the Project for this testing, it will be a time-consuming and tedious task to execute the test cases again and again.
Regression of GUI Application:
It is difficult to perform a GUI (Graphical User Interface) Regression test when the GUI structure is modified. The test cases written on the old GUI either become obsolete or need to be modified.
Re-using the regression test cases means GUI test cases are modified according to the new GUI. But this task becomes a cumbersome one if you have a large set of GUI test cases.
The Difference Between Regression And Re-testing:
Re-testing is done for the test cases which fail during the execution and the bug raised for the same has been fixed whereas Regression check is not limited to the bug fix as it covers other test cases as well to ensure that the bug fix has not impacted any other functionality of the Product.
Regression Test Plan Template (TOC)
- Document History
- References
- Regression Test Plan
3.1. Introduction
3.2. Purpose
3.3. Test Strategy
3.4. Features to be tested
3.5. Resource Requirement
3.5.1. Hardware Requirement
3.5.2. Software Requirement
3.6. Test Schedule
3.7. Change Request
3.8. Entry/Exit criteria
3.8.1. Entry Criteria for this Testing
3.8.2. Exit Criteria for this Testing
3.9. Assumption/Constraints
3.10. Test Cases
3.11. Risk /Assumptions
3.12. Tools - Approval/Acceptance
Let’s take a look at each of them in detail.
#1) Document History
Document history consists of a record of the first draft and all the updated ones in the below-given format.
| Version | Date | Author | Comment |
|---|---|---|---|
| 1 | DD/MM/YY | ABC | Approved |
| 2 | DD/MM/YY | ABC | Updated for the added feature |
#2) References
The References column keeps track of all the reference documents used or required for the Project while creating a test plan.
| No | Document | Location |
|---|---|---|
| 1 | SRS document | Shared drive |
#3) Regression Test Plan
3.1. Introduction
This document describes the change/update/enhancement in the Product to be tested and the approach used for this testing. All code changes, enhancements, updates, and added features are outlined to be tested. Test cases used for Unit Testing and Integration Testing can be used to create a test suite for Regression.
3.2. Purpose
The purpose of the Regression Test Plan is to describe what exactly and how testing would be performed to accomplish the results. Regression checks are done to ensure that no other functionality of the product is hampered because of the code change.
3.3. Test Strategy
Test Strategy describes the approach which will be used to perform this testing and that includes the technique that will be used, what will be the completion criteria, who will be performing which activity, who will write the test scripts, which regression tool will be used, steps to cover the risks like resource crunch, delay in production, etc.
3.4. Features to be tested
Features/components of the product to be tested are listed here. In regression, all the test cases are re-executed or the ones which affect the existing functionality are chosen depending on the fix/update or enhancement done.
3.5. Resource Requirement
3.5.1. Hardware Requirements:
Hardware Requirements can be identified here like computers, laptop, Modems, Mac book, Smartphone, etc.
3.5.2. Software Requirements:
Software Requirements are identified such as which Operating system and browsers will be required.
3.6. Test Schedule
The test schedule defines the estimated time for performing the testing activities.
For example, how many resources will perform a testing activity and that too in how much time?
3.7. Change Request
CR details are mentioned for which Regression will be performed.
| S.No | CR Description | Regression Test Suite |
|---|---|---|
| 1 | ||
| 2 |
3.8. Entry/Exit Criteria
3.8.1. Entry Criteria for this testing:
Entry criteria for the Product to start Regression check are defined.
For Example:
- Coding changes/enhancement/addition of new features should be completed.
- Regression test Plan should be approved.
3.8.2. Exit Criteria for this testing:
Here are the exit criteria for Regression as defined.
For Example:
- Regression testing should be completed.
- Any new critical bugs found during this testing should be closed.
- Test Report should be ready.
3.9. Test Cases
Regression Test cases are defined here.
3.10. Risk/Assumptions
Any risk & assumptions are identified and a contingency plan is prepared for the same.
3.11. Tools
Tools to be used in the Project are identified.
Such as:
- Automation tool
- Bug Reporting tool
#4) Approval/Acceptance
The names and designations of the people are listed here:
| Name | Approved/Rejected | Signature | Date |
|---|---|---|---|
Conclusion
Regression Testing is one of the important aspects as it helps to deliver a quality product by making sure that any change in the code whether it’s small or large does not affect the existing or old functionality.
A lot of automation tools are available for automating the regression test cases, however, a tool should be selected as per the Project requirement. The tool should have the ability to update the test suite as the Regression test suite needs to be updated frequently.
With that, we are wrapping this topic up and hope there will be much better clarity on the subject from now on.
YOU MAY BE INTERESTED IN
The Art of Software Testing: Beyond the Basics
Automation testing course in Pune