Generate a single report by merging multiple parallel runs

Learn how to generate a single build run report for builds that are run on multiple different CI machines or workers.

What is the use case?

When users use multiple machines (e.g. CI slaves) to run tests in parallel, multiple build runs are created on Test Observability. Similarly, in the absence of Test Observability, multiple test reports are generated and need to be manually merged to get a holistic view of the build run. For example, if ten machines are used to parallelize tests, ten different test reports will be generated without merging.

In such scenarios, a single report is more useful than multiple reports.

What are the benefits of generating a single report?

Here is a list of reasons why you should be generating one single report for builds run on multiple CI machines or workers:

• If you’re running hundreds of tests in your regression, you’d want to know what is the stability or passing percentage of your suite rather than separate statistics for builds run on individual CI machines or workers.
• You’d want to analyze common cause failures together. For example, if fifty tests have failed because a “Sign in” button is not visible, you’d want to see one single error causing fifty test failures for faster run verification.
• You’d want to track the entire run as one single instance for your metrics reporting rather than ten different instances, and many more.

How to generate a single report?

Test Observability offers the automatic functionality to merge reports into a single report through the use of environment variables. Follow these steps to use the functionality:

1. Set the environment variable BROWSERSTACK_BUILD_RUN_IDENTIFIER with a unique string. The value of the variable needs to be the same string across all the parallel machines where test runner jobs are running.
2. An easy way to do this could be to generate the value in the master job and propagate the value across all slave machines. E.g. export BROWSERSTACK_BUILD_RUN_IDENTIFIER=$RANDOM.
3. You could also set the value of the variable to date time or the master CI job number such that the number/value is the same across all the machines.