Rerun failed tests with Test Reporting & Analytics

Trigger reruns of failed tests from Test Reporting & Analytics and map them to previous runs of the same test case.

Automation engineers, especially those who run functional end-to-end tests, know the importance of rerunning test cases that fail on the first attempt. SDETs often rerun tests multiple times as part of the same CI job, or trigger the same job again with different parameters. Sometimes, the rerun is handled directly by the test framework.

Among the tests that you rerun, the common use case is to see only the latest status of the test. Test Reporting & Analytics solves this problem. You rerun your tests, and it maps each rerun to the previous runs of the same test case so that you see only the latest status.

The following scenarios are supported:

1. The framework automatically retries a failed test case.
2. You re-trigger a CI job with failed test cases.
3. The same CI job invokes the test runner again with the failed test cases.
4. You want to rerun test cases, a single test or multiple tests, during manual analysis.
5. You upload the results of a rerun as a JUnit XML or Allure report.

Map rerun CI jobs with existing build runs

Test Reporting & Analytics maps the test case runs with a previous run instance on a test rerun. If the test framework itself is rerunning the test, you don’t need to do anything and you see the retries of a test automatically.

If you rerun tests through approach 2 or 3, set the following environment variable in your runner environment just before you trigger the second and subsequent runs:

Rerun specific tests

You can also trigger reruns of specific tests while you’re manually analyzing the failures, right from within Test Reporting & Analytics.

To do this, configure the rerun setting and the CI integration, as shown in the following steps.

Configure rerun setting

Go to the Re-run settings to turn the Re-run configuration on or off. The configuration is on by default.

Configure CI integration for job triggers

After you turn on the rerun setting in the dashboard, the next step is to configure a CI integration so that BrowserStack can trigger CI jobs on your behalf when you choose to rerun specific tests.

Go to the Integrations page and configure one of the supported CI integrations. For the setup steps for your tool, see:

• Rerun tests on Jenkins
• Rerun tests on Azure
• Rerun tests on GitHub Actions
• Rerun tests on GitLab
• Rerun tests on CircleCI

Trigger reruns

After you complete the CI configuration, follow these steps to trigger reruns of failed test cases:

1. Go to the Test listing to view failed test cases.
2. Hover over the test name to display the action icons on the right, where the duration appears.
   
   
3. Click the rerun icon to open the rerun options.
   
   
   

Map reruns using JUnit XML or Allure reports

If you add tests to Test Reporting & Analytics by uploading JUnit XML reports or Allure reports, you can map the data to an earlier build run using one of the following parameters in the upload request:

Rerun a specific build run

Use buildIdentifier to map the tests to a particular build run. Set buildIdentifier to the exact value used in the earlier upload, and Test Reporting & Analytics maps the tests to that build run instead of creating a new one.

A buildIdentifier expires six hours after the first upload. After it expires, an upload with the same identifier starts a new build run instead of mapping to the existing one. If your reruns can happen later than six hours, use isRerun.

Rerun the latest build run

Use isRerun when you want the new upload to map to the most recent build run with the same buildName. Set isRerun to true and keep buildName unchanged from the run you want to map to. This parameter has no time limit, so you can use it for reruns that you trigger long after the original run.