Understanding System Testing

List of Tutorials:

• What is System Testing
• System vs end to end testing

evaluating an integrated hardware and software system to ensure that it satisfies predetermined standards.

Verification: The process of confirming through analysis and the presentation of impartial proof that certain conditions have been met.

Integration testing is the process of combining modules A and B, B and C, or A and C in an application that consists of three modules, A, B, and C. System testing is the process of integrating all three elements and testing the system as a whole.

My Experience

So do you think it will take that huge amount of time to test what you call System Testing, even after spending a lot of effort on Integration Testing?

The client we recently approached for the project was not convinced about the estimation we provided for each testing effort.

I had to chime in with an example:

Mike, I would like to elaborate on our efforts and the importance of system testing with an example.

Shoot, he replied.

System Testing Example

A car manufacturer does not manufacture the entire vehicle. Seats, steering, mirrors, breaks, cables, engines, automobile frames, wheels, and other parts are all created independently.

Unit testing is the process of independently testing each item once it has been manufactured to see if it is functioning as intended.

Integration testing is the process of checking that a component’s functionality has not been negatively impacted by assembly and that the two components are functioning as intended when each portion is put together with another element.

The car isn’t truly ready once all the components are put together and it’s ready.

The whole car needs to be checked for different aspects as per the requirements defined, like if the car can be driven smoothly, breaks, gears, and other functionality working properly. The car does not show any sign of tiredness after being driven for 2500 miles continuously.

The color of the car is generally accepted and liked, and the car can be driven on any kind of road like smooth and rough, sloppy and straight, etc. This whole effort of testing is called System Testing and it has nothing to do with integration testing.

The example performed as expected, and the client became convinced of the efforts required for the system test.

I narrated the example here to encourage the importance of this testing.

Approach

It is performed when Integration Testing is completed.

It is mainly a Black-box type testing. This testing evaluates the working of the system from a user point of view, with the help of a specification document. It does not require any internal knowledge of systems like the design or structure of the code.

It contains functional and non-functional areas of application/product.

Focus criteria:

It mainly focuses on:

1. External interfaces
2. Multiprogram and complex functionalities
3. Security
4. Recovery
5. Performance
6. Operator and user’s smooth interaction with the system
7. Installability
8. Documentation
9. Usability
10. Load/Stress

Why System Testing

#1) It is very important to complete a full test cycle and ST is the stage where it is done.

#2) ST is performed in an environment that is similar to the production environment and hence stakeholders can get a good idea of the user’s reaction.

#3) It helps to minimize after-deployment troubleshooting and support calls.

#4) In this STLC stage, Application Architecture, and Business requirements both are tested.

This testing is very important, and it plays a significant role in delivering a quality product to the customer.

Let’s see the importance of this testing through the below examples which include our day-to-day tasks:

• What if an online transaction fails after confirmation?
• What if an item placed in a cart of an online site does not allow one to place an order?
• What if in a Gmail account creating a new label gives an error on clicking the create tab?
• What if the system crashes when a load is increased on the system?
• What if the system crashes and cannot recover the data as desired?
• What if installing software on the system takes much more time than expected and at the end gives an error?
• What if a website response time increases much more than expected after enhancement?
• What if a website becomes too slow that the user is unable to book his/her travel ticket?

Above are just a few examples to show how System Testing would affect if not done properly.

All the above examples are just the result of either system testing not being performed or not being done properly. To ensure that the product works as per the requirements, we should test all the integrated modules.

Is This a White-box Or Black-box Testing?

System testing can be considered a black-box test technique.

Black box Testing technique does not require internal knowledge of the code, whereas the white box technique requires internal knowledge of the code.

While performing System testing, functional & non-functional, security, performance, and many other testing types are covered and they are tested using a black-box technique wherein the input is provided to the system and the output is verified. System internal knowledge is not required.

Black Box Technique:

How To Perform System Test?

It is a part of software testing and the Test Plan should always contain a specific space for this testing.

Clear requirements and expectations are necessary for testing the system as a whole, and the tester must also comprehend how the program is used in real time.

Also, the system’s functionality, performance, security, recoverability, and installability may be impacted by the most popular third-party tools, OS versions, flavors, and architecture.

Thus, a clear picture of how the program will be utilized and the types of problems it may encounter in real time can be useful while testing the system. A requirements document is just as crucial as comprehending the program, aside from that.

Clear and updated requirements documents can save the tester from several misunderstandings, assumptions, and questions.

In short, a pointed and crisp requirement document with the latest updates, along with an understanding of real-time application usage, can make ST more fruitful.

This testing is done in a planned and systematic manner.

Given below are the various steps involved in performing this testing:

• The very first step is to create a Test Plan.
• Create System Test Cases and test scripts.
• Prepare the test data required for this testing.
• Execute the system test cases and script.
• Report the bugs. Re-testing the bugs once fixed.
• Regression testing to verify the impact of the change in the code.
• Repetition of the testing cycle until the system is ready to be deployed.
• Sign off from the testing team.

What to test?

The points stated below are covered in this testing:

• End to End testing which includes verifying the interaction between all the components and along with the external peripherals to ensure if the system works fine in any of the scenarios is covered in this testing.
• It verifies that the input provided to the system provides the expected result.
• It verifies if all the functional & non–functional requirements are tested and if they work as expected or not.
• Ad-hoc and exploratory testing can be performed in this testing after scripted testing has been completed. Exploratory testing and ad-hoc testing help to unfold the bugs that cannot be found in scripted testing, as it gives freedom to the testers to test as their desire is based on their experience and intuition.

Advantages

There are several advantages:

• This testing includes end-to-end scenarios to test the system.
• This testing is done in the same environment as the Production environment, which helps to understand the user’s perspective and prevents the issues that can occur when the system goes live.
• If this testing is done systematically and properly, then it would help in mitigating the post-production issues.
• This testing tests both the application architecture and business requirements.

Entry/Exit Criteria

Let’s take a detailed look at the Entry/Exit criteria for System Test.

Entry Criteria:

• The system should have passed the exit criteria of Integration testing i.e. all the test cases should have been executed and there should be no critical or Priority P1, a P2 bug in an open state.
• Test Plan for this testing should be approved & signed off.
• Test cases/scenarios should be ready to be executed.
• Test scripts should be ready to be executed.
• All the non–functional requirements should be available and test cases for the same should have been created.
• The testing environment should be ready.

Exit Criteria:

• All the test cases should be executed.
• No critical or priority or security-related bugs should be in an open state.
• If any medium or low-priority bugs are in an open state, then they should be implemented with the acceptance of the customer.
• Exit Report should be submitted.

System Test Plan

Test Plan is a document that is used to describe the purpose, objective, and scope of a product to be developed. What has to be tested and what should not be tested, testing strategies, tools to be used, environment required, and every other detail is documented to proceed further with the testing.

The Test Plan helps to proceed with testing in a very systematic and strategic manner and that helps to avoid any risks or issues while testing is done.

System Test Plan covers the following points:

• Purpose & Objective is defined for this test.
• Scope (Features to be tested and features not to be tested are listed).
• Test Acceptance Criteria (Criteria on which the system will be accepted, i.e. mentioned points in acceptance criteria should be in the pass state).
• Entry/Exit criteria (define the criteria when system testing should start and when it should be considered as complete).
• Test Schedule (Estimation of testing to be completed at a specific time).
• Test Strategy (Includes testing techniques).
• Resources (Number of resources required for testing, their roles, resource availability, etc).
• Test Environment (Operating System, Browser, Platform).
• Test Cases (List of test cases to be executed).
• Assumptions (If any assumptions, they should be included in the Test Plan).

Procedure To Write System Test Cases

System test cases cover all the scenarios & use cases and also covers functional, non-functional, user interface, and security-related test cases. The test cases are written in the same way as they are written for functional testing.

System test cases include the below fields in the template:

• Test Case ID
• Test Suite name
• Description: Describes the test case to be executed.
• Steps: Step-by-step procedure to describe how to perform testing.
• Test Data: Dummy data is prepared to test the application.
• Expected Result: Expected result as per the requirement document is provided in this column.
• Actual Result: This column provides the result after executing the test case.
• Pass/Fail: Comparison in actual & expected results defines the Pass/fail criteria.
• Remarks

System Test Cases

Here are some sample test scenarios for an eCommerce Site:

1. If the site launches properly with all the relevant pages, features, and logo.
2. If the user can register/login to the site.
3. If the user can see products available, he can add products to his cart, can do payments, and can get confirmation via e-mail or SMS or call.
4. If the major functionality like searching, filtering, sorting, adding, changing, wishlist, etc work as expected.
5. If the number of users (defined as in the requirement document) can access the site simultaneously.
6. If the site launches properly in all major browsers and their latest versions.
7. If the transactions are being done on the site via a specific user are secure enough.
8. If the site launches properly on all the supported platforms like Windows, Linux, Mobile, etc.
9. If the user manual/guide return policy, privacy policy, and terms of using the site are available as a separate document and useful to any newbie or first-time user.
10. If the content of pages is properly aligned, well managed, and without spelling mistakes.
11. If session timeout is implemented and working as expected.
12. If a user is satisfied after using the site or user does not find it difficult to use the site.

Types Of System Testing

ST is called a superset of all types of testing, as all the major types of testing are covered in it. Although a focus on types of testing may vary based on product, organization processes, timeline, and requirements.

The overall it can be defined as below: 

Functionality Testing: To make sure that the functionality of the product is working as per the requirements defined within the capabilities of the system.

Recoverability Testing: To make sure how well the system recovers from various input errors and other failure situations.

Interoperability Testing: To make sure whether the system can operate well with third-party products or not.

Performance Testing: To make sure the system’s performance under the various condition, in terms of performance characteristics.

Scalability Testing: To make sure the system’s scaling abilities in various terms like user scaling, geographic scaling, and resource scaling.

Reliability Testing: To make sure the system can be operated for a longer duration without developing failures.

Regression Testing: To ensure the system’s stability as it passes through an integration of different subsystems and maintenance tasks.

Documentation Testing: To make sure that the system’s user guide and other help topics documents are correct and usable.

Security Testing: To make sure that the system does not allow unauthorized access to data and resources.

Usability Testing: To make sure that the system is easy to use, learn and operate.

More System Testing Types

#1) Graphical User Interface Testing (GUI)

GUI testing is used to check whether a system’s GUI functions as intended. In essence, a GUI is what a user sees when using an application. GUI testing includes testing a variety of elements, including menus, toolbars, dialog boxes, buttons, icons, checkboxes, list boxes, textboxes, and more.

#2) Compatibility Testing

Compatibility testing is done to ensure that the developed product is compatible with different browsers, Hardware Platforms, Operating Systems, and databases as per the requirement document.

#3) Exception Handling

Exception Handling Testing is performed to verify that even if an unexpected error occurs in the product, it should show the correct error message and does not let the application stop. It handles the exception in a way that the error is shown meanwhile the product recovers and allows the system to process the incorrect transaction.

#4) Volume Testing

Volume Testing is a type of non-functional testing wherein testing is done using a huge amount of data. For example, the Volume of data is increased in the database to verify the system performance.

#5) Stress Testing

Stress Testing is done by increasing the number of users (at the same time) on an application to the extent that the application breaks down. This is done to verify the point at which the application will break down.

#6) Sanity Testing

Sanity Testing is performed when the build is released with a change in the code or functionality or if any bug has been fixed. It verifies that the changes done have not affected the code. No other issue has occurred because of that and the system works as previously.

If in case any issue occurs, then the build is not accepted for further testing.

Thorough testing is not done for the build to save time & cost, as it rejects the build for an issue found. Sanity testing is done for the change done or for the fixed issue and not for the complete system.

#7) Smoke Testing

Smoke Testing is a testing that is performed on the build to verify if the build is further testable or not. It verifies that the build is stable to test and that all the critical functionalities are working fine. Smoke testing is done for the complete system, i.e. end to end testing is done.

#8) Exploratory Testing

Exploratory Testing as the name itself suggests, is all about exploring the application. No scripted testing is performed in exploratory testing. Test cases are written along with the testing. It focuses more on execution than planning.

Tester has the freedom to test on his own using his intuition, experience, and intellect. A tester can choose any feature to test first, i.e. randomly he can pick the feature to test, unlike the other techniques where the structural way is used to perform testing.

#9) Adhoc Testing

Adhoc Testing is informal testing where no documentation or planning is done to test the application. Tester tests the application without any test cases. A tester aims to break the application. The tester uses his experience, guess, and intuition to find the critical issues in the application.

#10) Installation Testing

Installation Testing is to verify if the software gets installed without any issues.

This is the most important part of testing, as the installation of the software is the very first interaction between the user and the product. The type of installation testing depends on various factors like operating system, platform, distribution of software, etc.

Test cases that can be included if an installation is done via the Internet:

• Bad network speed and broken connection.
• Firewall and security-related.
• Size and approximate time are taken.
• Concurrent installation/downloads.
• Insufficient memory
• Insufficient Space
• Aborted installation

#11) Maintenance Testing

Once the product goes live, the issue can occur in a live environment or some enhancement might be required in the product.

The product needs maintenance once it goes live and the maintenance team takes that care. The testing done for any issues or enhancement or migration to the hardware falls under maintenance testing.

What Is System Integration Testing?

It is a type of testing in which the system’s ability to maintain data integrity and operation in coordination with other systems in the same environment is being checked.

Example of System Integration Testing:

Let’s take the example of a well-known online ticket booking site – http://irctc.co.in.

This is a ticket booking facility; an online shopping facility that interacts with PayPal. Overall, you can consider it as A*B*C=R.

The online ticket booking, online shopping, and online payment option features may now be independently evaluated at the system level. Each of these features will then undergo integration testing. After then, a thorough testing of the entire system is required.Now on the system level, the online ticket booking facility, online shopping facility, and online payment option facility can be system tested independently, followed by check perform Integration tests for each of them. And then the whole system needs to be systematically tested.

So where does System Integration testing come into the picture?

The web portal http://Irctc.co.in is a combination of systems. You may perform tests at the same level (single system, the system of systems), but at each level, focus on different risks (integration problems, independent functionality).

• While testing the Online Ticket booking facility, you may verify if you can book tickets online. You may also consider integration problems. For example, the Ticket booking facility integrates the back-end with the front-end (UI). For example, how front-end behave when the database server is slow to respond?
• Testing of an online ticket booking facility with an online shopping facility. You may verify that the online shopping facility is available for the users logged into the system to book tickets online. You may also consider verification of integration in the online shopping facility. For example, if the user can select and buy a product without a hassle.
• Testing of online ticket booking facility’s integration with PayPal. You may verify whether, after booking tickets, money was transferred from your PayPal account to the Online Ticket Booking account. You may also consider the verification of integration in PayPal. For example, what if the system puts two entries in a database after debiting money for once only?

Difference between System Testing and System Integration Testing:

The main difference is:

• System Testing looks after a single system’s integrity with the relevant environment.
• System Integration Testing looks after multiple systems’ integrity with each other, being in the same environment.

Thus, the system test is the beginning of real testing, where you test a product as a whole and not a module/feature.

Difference Between System And Acceptance Testing

Given below are the major differences:

	System Testing	Acceptance Testing
1	System testing is the testing of a system as a whole. End to end testing is performed to verify that all the scenarios are working as expected.	Acceptance testing is done to verify if the product meets customer requirement.
2	System testing includes functional & non- functional testing and is performed by the testers.	Acceptance testing is functional testing and is performed by testers as well as a customer.
3	Testing is performed using test data created by the testers.	Real/Production data is used while performing acceptance testing.
4	A system as a whole is tested to check the functionality & Performance of the product.	Acceptance testing is done to verify that business requirement i.e. it solves the purpose what customer is looking for.
5	Defects found in the testing can be fixed.	Any defects found while acceptance testing is considered as a failure of the Product.
6	System and system integration testing are types for System testing.	Alpha and Beta testing come under acceptance testing.

Tips To Perform The System Test

1. Replicate real-time scenarios rather than doing ideal testing as the system is going to be used by an end-user and not by the trained tester.
2. Verify the system’s response in various terms, as the human does not like to wait or to see the wrong data.
3. Install and configure the system as per the documentation because that is what the end-user is going to do.
4. Involving people from different areas like business analysts, developers, testers, and customers can send in a better system.
5. Regular testing is the only way to make sure that the littlest change in the code to fix the bug has not inserted another critical bug into the system.

Conclusion

System testing is crucial, because if done incorrectly, serious problems could arise in the real world.

There are various aspects of a system that need to be confirmed. Any webpage would be a basic example. If the website isn’t tested comprehensively, consumers may find it extremely slow, or it may crash when several people log in at once.

It is not possible to test these features until the entire website has been tested.

YOU MAY BE INTERESTED IN

The Art of Software Testing: Beyond the Basics

Automation testing course in Pune

Automation testing in selenium

Mastering Software Testing: A Comprehensive Syllabus