“As the name implies, exploratory testing involves learning, designing, and executing tests all at once.We may argue that in this testing, test execution, analysis, design, and planning are all completed simultaneously.
Exploring the system and promoting a tester’s practical and real-time thinking are the goals of this testing..
Overview
To put it simply, exploratory testing entails designing test cases and running tests on the system or application being tested simultaneously. The tester will develop or document a test idea to provide guidance and investigate the system throughout testing in order to produce additional crucial, helpful, and practical tests for an application’s successful testing.
Planning is not necessary for this. Testers are always making decisions on what to do next. It is entirely dependent on the tester’s mentality.
Sometimes this testing can be more beneficial than the formal testing approach for finding some subtle defects that go missing in formal testing.
Consciously or unconsciously every tester would have done exploratory testing at some point in their career.
As we all know, a learner will learn better through hands-on experience rather than cramming the theory.
Same way, a tester will know the application better only while exploring and learning about all the functionality it provides by itself. It is always good to have a customer and business perspective while testing to ensure the successful testing of an application.
For Example, if you open a shopping website, you have a general idea that this shopping website will let you shop by selecting a product of your choice and then paying for the same.
During this process, you might learn that the website provides you with a virtual human look-alike which helps you in the product selection process. You also found that you can order several products for a home trial or that you can make payment through rewards points of some banks, etc.
As a tester, you not only need to verify whether a system is working as expected but also check if that system is not behaving in a way that is not expected.
Few things to remember while performing this testing:
- Your mission should be clear.
- Make sure to create notes and reports on what you are doing and how a system is behaving, which could be a potential bug.
- Learn, observe, and then come up with new test cases.
Exploratory Testing Examples
Example #1:
A home care service provider website with the following components:
- Login
- Services
- Cart
- Payment
- Order History
- Technician allotment
A general idea to start exploratory testing will be to log in or book a service.
How to Cover Test Cases?
In the above example, the idea is to start with functionality based on your knowledge. As you learn and observe more about the application, you can govern your next set of test cases.
Example# 2:
On one occasion, I worked on a minor project that included adding a new mutual fund to the program. It was my responsibility to test the application to ensure that consumers could purchase the new mutual fund and to verify that the valuation was accurate. I had two days to do my test.
Given the short turnaround time and the seriousness of the testing, I employed an exploratory testing methodology. Testing new features and looking for incompatibilities were my objectives.
My guiding principle for this test session was the aforementioned objective.
The following test cases evolved during this testing:
- Testing to make sure that the new Mutual fund has been added to the application.
- New MF is bought successfully.
- The valuation of the new MF is correct.
- Tried to buy a new MF for an existing portfolio.
- Can a new MF be added to all Portfolios?
- Impact of New MF on a valuation of existing.
- So in other test cases were evolved.
I prepared notes and reports during my testing to discuss my observation with the BA and the client involved.
The fundamental strategy of exploratory testing is to have a plan of attack. Start testing with your idea and improvise new test cases based on your knowledge and observation.
Example #3:
Exploratory Testing of the IRCTC Website
=> Click here to download the sample test cases of Exploratory Testing from the IRCTC website.
Testing Approach
- Make use of heuristics to guide testing.
- Test case execution and test case creation go hand in hand.
- Test cases keep on evolving based on tester observation and learning.
- Different testing techniques like Boundary value analysis, equivalence testing, etc. can be applied to ET.
- Session-based ET can be used to make it more structured and focused.
- Testers can branch out their ideas but never stray from your mission.
- ET testing does not use scripts, instead depends on the tester’s intuition, skill, and experience.
Benefits
Benefits of this testing include:
- Promote real-time thinking and helps in uncovering more defects.
- Promote use cases and scenario-based testing.
- Minimal documentation, maximum testing.
- Emphasis is more on learning and broadening the horizon of a tester.
- Avoid duplicate work.
- Useful when you want to audit other tester’s work.
Demerits
Demerits are enlisted below:
- Testing depends on the tester’s experience, skill, and knowledge.
- Require time to learn the application. Testers are more likely to miss if they know less about the application.
- Not appropriate for projects with long execution times.
Session-based Exploratory Testing
It can be quite challenging for testers to describe how much they have tested and on what basis while conducting exploratory testing.
The amount of time and effort required is hard to measure. Nonetheless, we must give team leads and managers KPIs, projections, and progress updates for each project. As they say, “you cannot manage it if you cannot quantify it.”
A time-based method for conducting this testing that aids in tracking and management is session-based testing. A specific time-box testing period free from phone calls, emails, messages, and other distractions is part of it.
Approach:
Testing tasks are divided into sessions.
The following are the components of session-based testing (SBT):
- Mission: The mission shouts out the purpose of the session and in a way provides the focus for the tester. It will also include a session time duration.
- Charter: Includes the scope of the testing. An agenda detailing the goals that need to be completed during the session.
Example of Test Charter for Login Functionality of Home Care Service website:
- Session: Pre-defined time-boxed testing session without any interruption. Each session can have the following duration:
- “Short” (60min)
- “Normal” (90min)
- “Long” (120 min)
- Session report: Include notes and a lightweight report to provide metrics to the leaders and managers. It gives details about the charter session remaining or done session setup time, scenario tested, the testing process, a list of bugs and the issues found, and other information for the metrics.
- Session De-brief: A short meeting or stand-up between the tester and Test Lead/Manager to review the test session findings.
Managers can get hands-on following metrics based on session reports:
- The number of sessions completed and remaining.
- The number of bugs reported.
- Time spent on session setup.
- Time spent on testing.
- Time spent on analyzing issues or problems.
- Features covered.
To summarize the above:
SBT provides improved time management for testing and enables accountability in exploratory testing. It also improves understanding of bug discovery and boosts productivity. It is an excellent method of giving managers and team leads metrics to monitor the project’s development.
Pair Based Exploratory Testing
Pair testing is a technique where two individuals share a PC and test the same item or feature of the program simultaneously. They are always exchanging ideas and opinions. One person controls the keyboard throughout this testing, while the other person makes test case suggestions and takes notes.
Effective communication between partners is always beneficial so that both parties understand what is being done and why. A pair is seen as a strong grouping when the testers’ strengths and weaknesses compliment one other.
Such pairing benefits both parties and each can learn something from their partner. It is also a good way to train new resources by pairing them with experienced resources.
Benefits of Pair Testing
- Helps a tester to focus on the task at hand.
- Encourage mutual trust and respect among partners.
- Brainstorming between paired testers usually, lead to more constructive ideas.
- Avoid tunnel vision.
- There is a lesser chance of others interrupting them.
Exploratory Testing Techniques
Tours:It is a straightforward method that enables a tester to imagine himself as a visitor to a city and employ his imagination. In this case, the city is an application to test, and the tourists are the testers. A visitor should have a plan with a specific objective in mind because it is quite difficult to explore the entire city without a lot of time and money.
A tourist can take the following tours:
- Guidebook tour – Testing highlighted features of the application. Use user-based scenarios.
- Exploring the history of the city – Test old features of an application.
- Money tour, which means making sure that all the critical features of the customer or client are tested and working successfully.
- Crime spree tour – Enter invalid input and test negative scenarios.
- Back alley tour – Test the least used features of the application.
- Boring tour – Spend minimum time on each screen of the application, fill in minimum fields, and take the shortest path. This will help with the default value and validation testing.
While taking a tour, you always have the choice of taking any route. You can navigate through the software and find a unique path to test the feature.
Below are some tips/tricks that you can use in ET:
- Divide the application into modules and bifurcate modules into different pages. Start your ET from the pages. This will give the right coverage.
- Make a checklist of all the features and put a checkmark when that is covered.
- Start with a basic scenario and then gradually enhance it to add more features to test it.
- Test all the input fields.
- Test for the error message
- Test all the negative scenarios.
- Check the GUI against the standards.
- Check the integration of the application with other external applications.
- Check for complex business logic.
- Try to do the ethical hacking of the application.
Factors that affect ET are as follows:
- The objective of the project
- Testing strategy
- The testing goal of a particular phase
- Available tools and facilities
- Testers role and skills
- Available time
- Management support
- Peer support
- Available resources (study materials, test conditions, etc)
- Clients interest
- Understandability of the product.
- The UI of the application
- The functionality of the application
- Previous test results
- Risks associated with the application
- Previous defects
- Recent changes
- Types of data to use for testing
- Type of user who will be using it
Instead of asking the testers what to run, we leave it to the testers’ judgment to decide what they want to test and how they want to test it.
Difference between Exploratory Testing and Ad-hoc Testing
Do not confuse ET with Ad-hoc testing.
- Ad-hoc testing refers to a process of unscripted, unplanned, and impromptu defect searching whereas exploratory testing is a thoughtful methodology for Ad-hoc testing.
- Ad-hoc testing is a hit-and-trial method of finding a bug whereas ET is not. In the ET approach, a tester learns about the system as they explore and eventually evolve the tests using the acquired knowledge.
- Ad-hoc testing is an unstructured activity whereas ET is somewhat a structured activity.
Exploratory Automated Testing (EAT)
Exploratory Automated Testing is a method that helps a tester streamline bug reporting & reproduction, snapshots gathering, and in preparation for future regression suits. It’s a process that combines automation testing with Exploratory testing.
There are two types of EAT approach:
- Passive EAT
- Active EAT
Passive EAT
A single tester or a pair of testers can conduct passive EAT. This practice often involves installing a tool on the testing resource’s computer that records and captures every action the resource performs.
With the exception of creating the test result based on the recorded session, passive EAT is identical to manually conducted ET in that it does not alter the way the tests are carried out. Later on, these test findings can be utilized to report and reenact recorded actions.
The installed video tool helps a tester with test case recording and defect reporting.
It also has a few other benefits like:
- Provides clear steps to reproduce the bugs.
- Reproducing defects is easier even when the defect reporter is not available.
- Do away with the conflicts between the testing and development team when an intermittent bug is reported.
- Helps in performance testing by getting the system response time at a specific point in time.
Here are a few other points to be taken into consideration before Passive EAT:
- It is advised to perform a pilot test before completely adapting the tool for Automated EAT. This is to ensure that the time required for re-designing of the test logs created during the test session is not more than test execution. If so, then the team needs to take a mutual decision on the following:
- If at all test automation is required for the particular project.
- If the tool being used needs to be changed.
- If the performance of the tool being used can be optimized.
- The tool used for performing automated EAT needs to be installed on every testing resource involved in testing. It is also a good idea to involve the developers which can be achieved by either giving the developers VPN or remote access to test machines or by installing the tool in the development environment.
- It is always a good idea to have the GUI object of the application organized in the test tool so that when the time comes to analyze the bug or an issue, the object is recognizable due to a meaningful name.
- It is a great practice to give a meaningful name to the GUI object used in AUT and keep it organized for later use.
Now, let’s move on to the second approach.
Active EAT
It is advisable to perform Active EAT with Pair Testing. In this approach, the keyword-driven testing is used in sync with Session testing. One tester creates the automated test script and the second tester executes the test scripts created by the first tester.
The creation of automation test scripts in this approach takes a different path than in conventional testing. Automated test scripts are made during testing and what has been discovered in the previous tests determines their design.
A closure phase is executed at the end of the testing session. And it should have the following tasks:
- Testers involved should swap roles so that the testing resource who created the test script has a chance to re-execute the scripts to confirm the reliability & robustness of the created suite.
- A brief description along with a few identifying characteristics should be provided for every automated test script.
- A criterion needs to be defined to identify which Automated test scripts can be used for the Regression tests.
Benefits of EAT
- At the start of each session, already created automated test scripts are executed thus enhancing the test coverage every time.
- Better bug reporting and documentation for defect reproduction.
- EAT provides enough evidence and documentation for a stakeholder to see the progress.
Types of Exploratory Testing
Given below are a few types of ET:
1) Freestyle ET:
Exploration of application in ad-hoc style.
In this type of ET, there are no rules, no account for coverage, etc. However, this type of testing is good when you need to familiarize yourself with the application quickly, when you want to verify the work of the other testers, and when you want to investigate a defect or want to do a quick smoke test.
2) Scenario-based ET:
As the name itself suggests, testing done is scenario-based. It starts with real user scenarios, end-to-end scenarios, or test scenarios. After initial testing, testers can inject variation as per their learning and observation.
Scenarios are like a generic guide for what to do during ET. Testers are encouraged to explore multiple possible paths while executing a scenario to ensure all possible paths to feature work. Testers should also ensure to gather as many scenarios as possible from different categories.
3) Strategy based ET:
Known testing techniques such as Boundary value analysis, equivalence technique, and risk-based technique are combined with exploratory testing. An experienced tester or a tester who is familiar with the application is appointed for this type of testing.
Agile Exploratory Testing
Even if you have not worked in an agile environment, I am sure you must have read or heard about it because of its growing popularity. Agile methodology has short sprints and tight deadlines which gives a team couple of weeks to finish planning, estimation, development, coding, testing, and release.
Exploratory testing becomes handy in such tight deadlines because in this testing approach emphasis is on the quick and useful result. Once you have understood the requirement, you can start testing based on your experience and knowledge.
Once you are familiarized with the application features and behavior, you can design more test cases to validate the application functionality and detect unplanned bugs. As it’s a freestyle testing approach, you do need to document everything. However, you need to maintain notes and a brief report on what you have tested, bugs and issues found, etc.
Merits of Exploratory in Agile
- Proving feedback to the developers as soon as possible.
- A broader variety of defects are uncovered.
- A diverse group of a resources such as a developer, tester, BA, and designers can perform ET as there are no scripted test cases and each brings a different perspective.
- Scouting done in ET helps in exploring new territories and exposing critical bugs.
- In the case of Iterative coding of an application, ET can focus on testing new features while automation does regression and backward compatibility testing.
- In case of unstable requirements, ET can help in testing new requirements within a limited time.
Points to Remember:
1. Requires different skills: The testers performing ET need to have good listening, reading, thinking, and reporting skills. Domain experience is required as there are no scripts and test cases.
2. Sometimes it’s difficult to report a bug: While in an ET flow, we may encounter a defect but we may not be able to reproduce it. This is because we are not tracking the testing steps and we may forget the exact steps to reproduce that issue.
3. Can be done as a recreation activity: I do ET when I want a break from my regular test execution cycle. However many teams have ET as a separate phase of their testing cycle.
4. It can be done for all testing phases: We can implement ET before the beginning of any testing phase. You can perform ET even before the functional testing phase.
5. Rapid feedback: ET requires rapid feedback on the issues and any anomalies encountered.
6. Critical Thinking and diverse ideas: This testing requires critical thinking. Testers should be able to reproduce, review and logically express their ideas. A tester can implement her experience in the various technologies and domains they worked on.
How to Think Beyond Traditional Testing Boundaries in Exploratory Testing
“I really appreciate your concern for the product and making us helpful in understanding end-user perspective. It’s going to be very helpful. Thanks for the good work and keep it up!!!”
This was the last e-mail of an email chain with 21 emails from our client. It was midnight, and our product release was delayed due to a critical bug we found. You might think, what is new in that? It may happen many times. But, this was really different as the critical bug we reported was not a result of any documented test case.
After completing regression testing for the last time that evening, I was just playing with the product. What does that mean? You are free to do what you are not supposed to do. Based on my experience and project knowledge, I had some ideas on how to test the product apart from our typical test repository, called Exploratory Testing.
The exploratory testing done found a critical bug related to a server hang issue while doing something unexpected.
Being a fan of exploratory testing, I love to explore the product in different ways. For me, the definition of software is:
“It should do what it is supposed to do, and it should not do what it is not supposed to do.”
You are an incomplete tester if you restrict your testing to see if goods that should work are functioning. Actually, the completion of recorded regression testing and the updating of results marks the beginning of a tester’s life. It makes a significant difference to view things from many angles and comprehend end-user needs in various contexts. So let’s work together to discover how that change can be created today:
How to look at a product from different perspectives?
#1. Understand customer/end-user
The main goal of software testing is to evaluate the product’s quality in relation to user satisfaction. How can you determine a customer’s perspective? The answer is straightforward: you must be the client. Alright, let me to correct you. It won’t be sufficient to be a customer.
You must ascertain the customer’s preferred method of handling the merchandise. No two consumers will make the same dish even if they purchased the identical ingredients. Indeed, the product we create or supply serves as a raw material for our clients’ enterprises, and when they use it, their perspective changes.
As software testers, we must examine the product’s intent rather than its features or objects.
Let me give you some real-life practical examples:
- Scissors were never limited to cutting paper only. Cutting is the purpose and not the paper (object).
- Cell phones were never limited to only calling, but “able to call” has always been the basic purpose.
- Storage boxes are used to store, but the safety of the material stored is as important as storage.
Understanding stakeholders and a wide range of their expectations should be the baseline of exploratory testing.
#2. A mindset
While looking for (let’s say) a job employment ad, do you see that jackpot and in between the pages with the bold font? Most of us do not (believe me, it’s true). Because we have instructed our minds to look for what is useful or to be checked. Anything else is of no use, so the mind denies us recognizing it.
Open your mind, and do not set any expectations when you start exploring a product. Always remember, it’s not OK if the product is doing what it is supposed to do. It is also important that it should not do what it is not supposed to do.
I remember one classic example:
In Linux, the “cat” command is used to check a file’s content, and the “ls” command is used to examine a directory’s content. After five years of working with Linux and software testing, I never considered using cat since I had a predetermined idea that I needed to use “ls” if I required dir content.
Although that was successful, the expectation was incorrect because the product was not designed to act in that manner. The system failed after one of our clients, who was not familiar with Linux, accidentally typed cat . This mentality cost us money.
Always be ready to make mistakes with the software because that is what the end user is going to do. To test the software, you have been trained, but the end-user will not be as trained as you or he/she will not be as much of a technical expert as you. Also, he/she will do anything with the software when they are in trouble.
Think about those scenarios, and provide testing feedback. The life of the software and yours (as a tester) will rock.
#3. Know the competitors
While testing any software application for your client, did you ever attempt to know and understand other software with the same purpose? Did you ever suggest some useful functionality you observed in a competitor’s product? It does not fall in our job description, which is the typical answer. But do you know the benefits of doing it?
Here are a few real-life examples to make you understand the point:
- Don’t you like the designer who not only stitches your dress but also provides you input about matching accessories most?
- Don’t you like the pizza brand that not only makes great pizzas but home delivers on time most?
- Don’t you like the photographer who not only takes good photographs but suggests different kinds of frames for the photoshoot?
Everyone wants to have something extra for what they pay for. Our analysis of competitive software can work the same way for us. The customer always likes to hear valuable suggestions – mainly comparative suggestions to make the product more useful or marketable.
Also, this kind of comparison and analysis of the same range of products makes our analysis more powerful, and eventually, we create a treasure to which we can go back at any moment and find something useful.
Conclusion
Exploratory testing is a very successful approach even if it is not a conventional one.
In order to find errors, it pushes testers to think creatively and motivates them to create realistic, real-time test scenarios. Its freestyle nature gives it an edge over other testing techniques, and it can be used anywhere, whether the project is Waterfall, Agile, or any other project that requires less documentation.
The success of exploratory testing depends on a number of intangible elements, including a tester’s skill, aptitude for crafting engaging test cases, experience, and ability to trust their gut.
To maintain a good balance between exploratory and scripted, it is important to remember that ET is an adaptable process rather than a necessary one or regular testing.
YOU MAY LIKE THIS
Verification and Validation in Software Testing: A Comprehensive Guide
Software Testing Lab Experiments: A Comprehensive Guide
Launch Your HR Career in Pune success: Mastering SAP SuccessFactors course with Expert-Led Training