Rerun failed tests with Test Reporting & Analytics

Trigger reruns of failed tests from Test Reporting & Analytics and map them to previous runs of the same test case.

Automation engineers, especially those who run functional end-to-end tests, know the importance of rerunning test cases that fail on the first attempt. SDETs often rerun tests multiple times as part of the same CI job, or trigger the same job again with different parameters. Sometimes, the rerun is handled directly by the test framework.

Among the tests that you rerun, the common use case is to see only the latest status of the test. Test Reporting & Analytics solves this problem as you can not only rerun tests but also automatically map the reruns with previous runs of the same test case and reflect only the latest status of the test.

The following scenarios are supported:

1. The framework automatically retries a failed test case.
2. You re-trigger a CI job with failed test cases.
3. The same CI job invokes the test runner again with the failed test cases.
4. You want to rerun test cases, a single test or multiple tests, during manual analysis.

Map rerun CI jobs with existing build runs

Test Reporting & Analytics maps the test case runs with a previous run instance on a test rerun. If the test framework itself is rerunning the test, you don’t need to do anything and you see the retries of a test automatically.

If you happen to be rerunning tests either through approach (2) or (3) as mentioned above, you need to set the following environment variable in your runner environment just before triggering the second and subsequent runs:

Rerun specific tests

You can also trigger reruns of specific tests while you’re manually analyzing the failures, right from within Test Reporting & Analytics.

To do this, configure the rerun setting and the CI integration, as shown in the following steps.

Configure rerun setting

Go to the Re-run settings to turn the Re-run configuration on or off. The configuration is on by default.

Configure CI integration for job triggers

After you turn on the rerun setting in the dashboard, the next step is to configure a CI integration so that BrowserStack can trigger CI jobs on your behalf when you choose to rerun specific tests.

Go to the Integrations page and configure one of the supported CI integrations. For the setup steps for your tool, see:

• Rerun tests on Jenkins
• Rerun tests on Azure
• Rerun tests on GitHub Actions
• Rerun tests on GitLab
• Rerun tests on CircleCI

Trigger reruns

After you complete the CI configuration, follow these steps to trigger reruns of failed test cases:

1. Go to the Test listing to view failed test cases.
2. Hover over the test name to display the action icons on the right, where the duration appears.
   
   
3. Click the rerun icon to open the rerun options.
   
   
   

Map reruns using JUnit XML reports

You can also map reruns of test cases with the previous run instances through JUnit XML reports, if you use JUnit XML reports to add data to Test Reporting & Analytics. To do this, use the same buildIdentifier across uploads that belong to the same build run.