Compare build runs

Learn to compare different build runs in your test suites.

As part of build run verification, QA engineers often need to compare the current build with a specific previous run of the same build or some other build run thoroughly before signing off. This is done to eliminate false positives and benchmark various metrics like performance against a stable build. This comparison however is a time-consuming process that requires users to open the builds under scrutiny side-by-side and go through them manually.

Test Reporting & Analytics provides an efficient and user-friendly option for comparing build runs in detail. An automated approach that highlights key differences—such as newly failed tests, passing tests, and changes in failure reasons—saves time and improves accuracy.

Build comparison in Test Reporting & Analytics is especially useful when a build run shows anomalies, such as an unusually high number of test failures.

How to compare build runs

You can initiate a build runs comparison from two places:

• The Build Runs page.
• Within a particular build.

How to compare build runs from the Build Runs page

Build Runs in Test Reporting & Analytics lists all your builds in your test suite. Follow these steps to compare build runs from the Build Runs page:

1. Open Build Runs.
2. Optionally, use filters and search options to quickly find the exact build run you want to compare.
3. Select the checkboxes next to any two build runs you want to compare and click Compare builds.
   .

A new tab opens with the comparison of the two build runs.

You can click Swap Builds if you want to interchange the Target Build and the Baseline Build.

You can further edit the builds being compared even when you view the build comparison. Use the edit button on target build or baseline build to add a new build to the comparison.

How to compare build runs from within a particular build

Follow these steps to compare build runs from within a particular build.

1. Click a build from Build Runs to view it in detail.
2. Click Compare this build.
   
   
3. Find the build run you want to compare with using the filtered views and search options.
   
4. Select the build run.

A new tab opens with the comparison of the two build runs.

You can click Swap Builds if you want to interchange the Target Build and the Baseline Build.

You can further edit the builds being compared even when you view the build comparison. Use the edit button on target build or baseline build to add a new build to the comparison.

Target build and baseline build

The two build runs compared are named as follows:

• Target Build: The build run you want to compare with another reference build run.
• Baseline Build: The reference build run against which you want to compare a build run with.

The following table summarizes how the builds are named in the two flows:

View comparison of build runs

When you compare two build runs, a new tab opens with a comparison view.

The build comparison report can be understood better by splitting it into three broad parts:

• Summary and metadata
• Highlights
• Detailed comparison

Summary and metadata

The comparison report begins with a side-by-side listing of the summary of the two build runs.

You can view the following details of the target build and the baseline build:

• Name of the build run
• Build tags, if any
• A quick summary of the test statuses
• Build status
• Owner
• Time of build run
• CI information

You can click View metadata to view further details. View metadata opens a modal comparing the metadata of target build and baseline build. The differences in metadata are highlighted in yellow.

Highlights

The second section highlights the key takeaways of the comparison:

• Newly failed: Number of tests that failed in the target build which ran fine in the baseline build.
• Failed with different reason: Number of tests that failed in the target build due to a different error compared to the one in the baseline build.
• Now Passing: Number of tests that ran successfully in the target build that had failed in the baseline build.
• Additional Tests: Number of tests in the target build which were not present in the baseline build.
• Dropped Tests: Number of tests that were present in the baseline build which are not present in the target build.
• Unchanged Tests: Number of tests that are common between the baseline build and the target build.

You can click the values given in the highlights to view those tests.

All test statuses other than passed are treated as failed in the build run comparison. For example, if a test with passed in the baseline build is marked as unknown in the target build, it is categorized as Newly failed.

Filter or search tests in comparison

You can search for a test or apply filters in the build comparison view. When you use these filters, only the tests that meet the criteria are included in the comparison.

Detailed comparison

In the last section, you can view a detailed view of the highlights. This is a list of all tests sorted according to folders and compared side-by-side.

• Newly failed: View all tests that failed in the target build which ran fine in the base build.
• Failed with different reason: View all tests that failed due to a different error compared to the one in the baseline build.
• Now Passing: View all tests that ran successfully in the target build which had failed in the baseline build.
• Additional Tests: View all tests in the target build that were not present in the baseline build.
• Dropped Tests: View all tests that were present in the baseline build which are not present in the target build.
• Unchanged Tests: View all tests with the same status between the baseline build and the target build.