How to write a good Test Summary Report?

Test Reports were originally started to be used in Waterfall Models. Still, nowadays, teams have started adopting it in Agile Development processes too, which has proved to be of great help.

A test report summary contains all the details of the testing process, what was tested, when was it tested, how it was tested, and the environments where it was tested. Compared to those created in the Waterfall versions, the test summary report in Agile development processes is less formal and more focused on the results.

Let’s discuss what a test summary report is and how it can help in software development lifecycle processes.

What is a Test Summary Report?

The definition of a Test Summary is as simple as the name suggests. It is also known as a Test Closure Report. It provides the relevant stakeholders with a detailed account of the overall test results and defects. It aims to summarize the results of the entire testing process formally.

Test Summary Report is an important document that is prepared at the end of a Testing project, or rather after the Testing cycle has been completed. The prime objective of this document is to explain the details of the Testing activities performed for the Project, to the respective stakeholders like Senior Management, Clients, etc. It also portrays the overall quality level of the application. 

While the Daily Status Reports only deliver the daily test status and results to the stakeholders, Test Summary Report on the other hand provides a consolidated report on the overall testing activities performed in the cycle.

Benefits of Test Report

The typical benefits of a test report include:

Pillars of a good Test Summary Report test in Software Testing

The pillars of a good test summary report are:

• Specification: You don’t need to write a very long test summary report. It should have all the test result specifications concisely and to the point.
• Standard: The test summary report should follow a standard template as it is easy for stakeholders to review and understand.
• Clarity: The information captured in the test report should be clear. The report should reflect brevity and clarity.
• Detail: The report should provide detailed information about the testing activities wherever necessary. The information though concise should not be abstract as it won’t help the stakeholders in drawing a clear picture of it.

When to create a Test Summary Report?

A test summary report is ideally created at the end of testing cycles, so it can also include details about regression tests. However, there should be enough time after submitting the report and before the product is shipped to the customers. The intention here is to help the client and the stakeholders with the information on the overall health of the testing cycle and the application being tested so that any corrective action can be taken if necessary. 

What to include in a Test Summary Report?

An informative test summary report should be concise and relevant. Several examples of Test Summary Report templates are available online, but all of them might not be applicable in your case. Therefore, it is vital to customize our report according to the nature of our test project after doing a proper analysis. 

Below is a list of what should be included:

1. Test Objective – Include the purpose of the testing, this shows that the test object and the requirement were clear with the testing team.
2. Areas Covered – Include the regions and the functionalities of the product that were covered in testing. It need not capture every test scenario in detail but should cover all the areas at a high level. 
3. Areas Not Covered – It is essential to capture the product areas that were not covered in testing. Any areas not tested can raise the alarm at the client’s end, so we should ensure that anything left untested is noted and expectations around it are set accordingly. Also, ensure that each point has an associated reason, for example, limitation of access to the availability of devices.
4. Testing Approach – This is important to cover since it indicates what and how the testing was performed. Ensure to provide details of the steps taken and the testing approaches adopted to achieve the task.
5. Defect Report – Though it usually gets captured already in the bug report, including it in the test summary report can give an additional advantage to your report. 
6. Platform Details – Nowadays, products are tested across multiple platforms. With the growing demand, testing is not just limited to multiple devices or browsers but different versions. So, let’s include details of every platform and environment on which the product was tested.
7. Overall Summary – This section is mainly for us to provide feedback on the application’s overall status under test. It should inform the client about the critical issues discovered and how many are still open so that they can estimate how close they are to shipping the product to the customer.

Steps to create a good test summary report

Now that we have learned about Test Summary reports, let us try to create a sample test report.

We will create a sample test report on ‘XYZ online travel company’.

Step 1: Capture the purpose of the document

In this step, let’s briefly describe the document’s purpose.

For Example, This document captures the various activities performed as part of Testing the ‘XYZ’ online travel company application.

Step 2: Capture an overview of the product in test

In this step, let’s briefly capture an overview of the application tested.

For Example, ‘XYZ online travel company’ is an online travel services application that offers services for booking airline tickets, domestic and international holiday packages, hotel reservations, rail, and bus tickets. Several modules like Registration, Booking, Payment, and Reports are integrated to fulfill the purpose.

Step 3: Capture the Testing Scope

In this step, we should capture the testing scope of the application, which means the areas/modules that are In Scope, Out of Scope and also the areas that were not tested due to any constraints or dependencies.

For Example, We can capture the areas of the product as shown below.

• In-Scope: Functional Testing for the following modules are in Scope of Testing
  • Registration Of Users
  • Booking of Tickets
  • Booking of hotel packages
  • Payment
  • Registration confirmation
• Out of Scope: Concurrency and multi-tenant user testing are out of scope in this release.
• Areas not tested: The user registration page with the field values in mixed cases e.g., XYZ and cancellation of multiple tickets together, could not be covered due to known issues.

Step 4: Capture the Metrics

Test Metrics help us to understand the test execution results, status of the cases as well as defects, etc. 

Also, Charts or Graphs can be attached for better representation to capture Defect distribution- function or module-wise, or severity-wise.

In the below chart, we have captured the total no of cases that were planned, the no of cases that were executed, the no of passed and failed cases, the no of defects identified, etc.

• No. of test cases planned 
• No. of test cases executed
• No. of test cases passed
• No. of test cases failed

• No of defects identified with their Status & Severity

Step 5: Capture the types of testing performed

In this step, capture the various Testing types performed for the product. This ensures that the application is being tested properly. We can also capture the details if multiple rounds of testing are done. For example,

1. Smoke Testing
2. Regression Testing
3. Integration Testing

For Example,

Smoke or Sanity Testing

Smoke or Sanity tests were performed whenever a new build was deployed into the test environment to ensure that the major functionalities of the application are working fine. If the test results were up to mark, the build was accepted and promoted to other test environments for the QA team to start testing.

Regression Testing

• Regression Testing was done once the smoke results were good on any new build deployed in the test environments.
• Regression Results were analyzed to ensure that the entire functionality works fine with the new build deployed.
• Regression Runs also covered the new test cases that were added for the newer areas in the test or any defect fixes that went in.

Integration Testing

This was performed to ensure that the entire application, integrated with all other functionalities, works as expected as intended without any errors.

Step 6: Capture the Test Environment and the Tools used

In this step, capture the details about the Test Environment in which the testing activities mainly were carried out, Application URL, Database version, etc. Also, capture the details of any tools used for Bug Management, like JIRA or Selenium for automation.

For Example,

Step 7: Learnings during the test activity

Utilize this section to describe the critical issues faced and the solutions implemented to get past those problem areas during the testing. These learnings made will help during the next Testing phase, and hence it is important to capture the details about them.

For Example,

Step 8: Recommendations or Suggestions, if any

In this section, we can mention any recommendations or suggestions for the relevant stakeholders. These suggestions can help plan for the next cycle and help the involved teams.

For Example,

• Administrator user details could be shared with the team across all time zones, so there is no dependency on the Onsite counterparts of the testing team for their usual day-to-day activities.
• The Development Team should conduct demos for any new feature introduced in the application to the entire Test team. This will help them to get an overview of the feature before the testing starts.

Step 9: Exit Criteria

Exit Criteria is defined as Test Completion when certain conditions are fulfilled like

• All test cases that were planned are executed successfully
• All critical issues are closed
• Any other open issues have an action planned and are targeted for the next release cycle.

Step 10: Sign Off

This section captures whether the Testing team gives a Green Signal for the application to Go Live and if the Exit Criteria were fulfilled. Suppose the Exit Criteria are not fulfilled completely. In that case, the relevant areas of concern can be highlighted and the decision can be left further with the Senior Management and other stakeholders if the application can Go Live or not.

How to share test summary reports within the team?

Once the test summary reports are designed, it is essential to share them with the stakeholders and clients and the team. With the test summary report, the team members will get an overview of the entire testing cycle which will help them in improving further. The reports can be of great help for newcomers as well. 

You can leverage BrowserStack integration with Slack to share these test summary reports daily within the team in an automated way. Integrating Slack with BrowserStack can help you to debug your failed tests directly from Slack and obtain a summary of all your builds executed during the day.

Try BrowserStack for free