How to write a good Test Summary Report?
By The Nerdy Geek, Community Contributor - July 14, 2022
Test Reports were originally started to be used in Waterfall Models but nowadays teams have started adopting it in Agile Development processes too and it has proved to be of great help.
A test report summary contains all the details of the testing process, what was tested, when was it tested, how it was tested, and the environments where it was tested. As compared to those created in the Waterfall versions, the test summary report in Agile development processes is less formal and is more focused on the results.
Let’s discuss what is a test summary report and how it can help in software development lifecycle processes.
Also Read: Difference between CI/CD, Agile and DevOps
What is a Test Summary Report?
The definition of a Test Summary is as simple as the name suggests. It is also known as a Test Closure Report. It provides the relevant stakeholders with a detailed account of the overall test results and the defects found. It aims to formally summarize the results of the entire testing process.
Test Summary Report is an important document that is prepared at the end of a Testing project, or rather after the Testing cycle has been completed. The prime objective of this document is to explain the details of the Testing activities performed for the Project, to the respective stakeholders like Senior Management, Clients, etc. It also portrays the overall quality level of the application.
While the Daily Status Reports only deliver the daily test status and results to the stakeholders, Test Summary Report on the other hand provides a consolidated report on the overall testing activities performed in the cycle.
Read More: How to set goals for Software Quality
Benefits of Test Report
The typical benefits of a test report include:
Pillars of a good Test Summary Report
The pillars of a good test summary report are:
- Specification: You don’t really need to write a very long test summary report. It should have all the test result specifications in a concise manner and to the point.
- Standard: The test summary report should follow a standard template as it is easy for stakeholders to review and understand.
- Clarity: The information captured in the test report should be clear. The report should reflect brevity and clarity.
- Detail: The report should provide detailed information about the testing activities wherever necessary. The information though concise should not be abstract as it won’t help the stakeholders in drawing a clear picture of it.
When to create a Test Summary Report?
A test summary report is ideally created at the end of testing cycles, so it can also include details about regression tests. However, there should be enough time after the report is submitted and before the product is to be shipped to the customers. The intention here is to help the client and the stakeholders with the information on the overall health of the testing cycle and the application being tested so that any corrective action can be taken if necessary.
What to include in a Test Summary Report?
An informative test summary report should be concise and relevant. There are several examples of Test Summary Report templates that are available online but all of them might not be applicable in your case. Therefore, it is vital to customize our report according to the nature of our test project after doing a proper analysis.
Below is a list of what should be included:
- Test Objective – Include the purpose of the testing, this shows that the test object and the requirement were clear with the testing team.
- Areas Covered – Include the areas and the functionalities of the product that were covered in testing. It need not capture each and every test scenario in detail but should cover all the areas at a high level.
- Areas Not Covered – It is very essential to capture the areas of the product that were not covered in testing. Any areas not tested can raise an alarm at the client’s end, so we should ensure that anything that has been left untested is noted and expectations around it are set accordingly. Also, ensure that each point has an associated reason, for example, limitation of access to the availability of devices.
- Testing Approach – This is important to cover since it indicates what and how the testing was performed. Ensure to provide details of the steps taken and the types of testing approaches that were adopted to achieve the task.
- Defect Report – Defect Report, though it usually gets captured already in the bug report, having it included in the test summary report can give an additional advantage to your report.
- Platform Details – Nowadays, products are tested across multiple platforms. With the growing demand, testing is not just limited to multiple devices or browsers but different versions as well. So, let’s ensure to include details of every single platform and environment on which the product was tested.
- Overall Summary – This section is mainly for us to provide our feedback on the overall status of the application under test. It should inform the client about the critical issues that were discovered and how many are still open so that they can estimate how close they are to shipping the product to the customer.
Read More: Test Case Prioritization: A Detailed Guide
Steps to create a good test summary report
Now that we have learned about Test Summary reports, let us try to create a sample test report.
We will create a sample test report on ‘XYZ online travel company’.
Step 1: Capture the purpose of the document
In this step, let’s capture a short description of the purpose of the document.
For Example, This document captures the various activities that were performed as part of the Testing of the ‘XYZ’ online travel company application.
Step 2: Capture an overview of the product in test
In this step, let’s capture an overview of the application tested in brief.
For Example, ‘XYZ online travel company’ is an online travel services application that offers services for booking airline tickets, domestic and international holiday packages, hotel reservations, rail, and bus tickets. There are several modules like Registration, Booking, Payment, and Reports which are integrated to fulfill the purpose.
Step 3: Capture the Testing Scope
In this step, we should capture the testing scope of the application, which means that the areas/modules that are In Scope, Out of Scope and also the areas that were not tested due to any constraints or dependencies.
For Example, We can capture the areas of the product as shown below.
- In-Scope: Functional Testing for the following modules are in Scope of Testing
- Registration Of Users
- Booking of Tickets
- Booking of hotel packages
- Registration confirmation
- Out of Scope: Concurrency and multi-tenant user testing are out of scope in this release.
- Areas not tested: The user registration page with the field values in mixed cases for eg, XYZ and cancellation of multiple tickets together could not be covered due to known issues.
Step 4: Capture the Metrics
Test Metrics help us to understand the test execution results, status of the cases as well as defects, etc.
Also, Charts or Graphs can be attached for better representation to capture Defect distribution- function or module wise, or severity wise.
In the below chart, we have captured the total no of cases that were planned, the no of cases that were executed, the no of passed and failed cases, the no of defects identified, etc.
- No. of test cases planned
- No. of test cases executed
- No. of test cases passed
- No. of test cases failed
- No of defects identified with their Status & Severity
Step 5: Capture the types of testing performed
In this step, capture the various types of Testing that were performed for the product. This ensures that the application is being tested properly. We can also capture the details if multiple rounds of testing were done. For example,
- Smoke Testing
- Regression Testing
- Integration Testing
Smoke or Sanity Testing
Smoke or Sanity tests were performed whenever a new build was deployed into the test environment to ensure that the major functionalities of the application are working fine. If the test results were up to mark, the build was accepted and promoted to other test environments for the QA team to start testing.
- Regression Testing was done once the smoke results were good on any new build that was deployed in the test environments.
- Regression Results were analyzed to ensure that the entire functionality works fine with the new build deployed.
- Regression Runs also covered the new test cases that were added for the newer areas in the test or any defect fixes that went in.
This was performed to make sure that the entire application on a whole, integrated with all other functionalities, works as expected as intended without any errors.
Step 6: Capture the Test Environment and the Tools used
In this step, capture the details about the Test Environment in which the testing activities were mostly carried out, Application URL, Database version, etc. Also, capture the details of any tools that were used for Bug Management like JIRA or Selenium for automation.
Step 7: Learnings during the test activity
Utilize this section to describe the critical issues that were faced and the solutions that were implemented to get past those problem areas during the testing. These learnings made will help during the next Testing phase, and hence it is important to capture the details about them.
Step 8: Recommendations or Suggestions, if any
In this section, we can mention any recommendations or suggestions for the relevant stakeholders, if any. These suggestions can help in the planning for the next cycle and can help the involved teams.
- Administrator user details could be shared with the team across all time zones, so that there is no dependency on the Onsite counterparts of the testing team for their usual day-to-day activities.
- There should be demos conducted by the Development Team for any new feature introduced in the application, to the entire Test team. This will help them to get an overview of the feature before the testing starts.
Step 9: Exit Criteria
Exit Criteria is defined as Test Completion when certain conditions are fulfilled like
- All test cases that were planned are executed successfully
- All critical issues are closed
- Any other open issues have an action planned and are targeted for the next release cycle.
Step 10: Sign Off
This section captures whether the Testing team gives a Green Signal for the application to Go Live and if the Exit Criteria were fulfilled. If the Exit Criteria are not fulfilled completely, then the relevant areas of concern can be highlighted and the decision can be left further with the Senior Management and other stakeholders if the application can Go Live or not.
How to share test summary reports within the team?
Once the test summary reports are designed, it is important to share them not just with the stakeholders and clients but within the team as well. With the test summary report, the team members will get an overview of the entire testing cycle which will help them in improving further. The reports can be of great help for the newcomers as well.
You can leverage BrowserStack integration with Slack to share these test summary reports on a daily basis within the team in an automated way. Integrating Slack with BrowserStack can help you to debug your failed tests directly from Slack and obtain a summary of all your builds executed during the day.