Skip to main content

Re-run tests with Test Observability

Learn how you can trigger re-runs of your tests

Supported scenarios for re-runs

Automation engineers especially the ones who run functional end-to-end tests, know the importance of re-running test cases that fail on the first attempt. SDETs often re-run tests multiple times as part of the same CI job or re-trigger the same job with different params or sometimes the re-run is handled directly by the test framework.

Among the tests that re-run, the common use case is to see only the latest status of the test. We’ve solved that problem in Test Observability wherein we not only allow you to re-run tests but also automatically map the re-runs with previous run of the same test case and reflect only the latest status of the test.

The following scenarios are supported:

  1. Framework automatically re-tries a failed test case.
  2. You’re re-triggering the CI job with failed test cases.
  3. The same CI job invokes the test runner again with the failed test cases.
  4. You want to re-run test cases (a single test or even multiple) during manual analysis.

Map re-run CI jobs with existing build runs

Test Observability maps the test case runs with a previous run instance if it happens to be a re-run. If the test framework itself is re-running the test, you need not do anything and you’d see the retries of a test automatically.

If you happen to be re-running tests either through approach (2) or (3) as mentioned above, you need to set the following environment variable in your runner environment just before triggering the second and subsequent runs:

Copy icon Copy snippet

BROWSERSTACK_RERUN=true environment variable will ensure that all tests that run with this variable set, are treated as re-runs of the immediately preceding run. Ensure to set the variable to false before the next fresh build run which is not to be treated as a re-run.

Re-run specific tests

You can also trigger re-run of specific tests while you’re manually analyzing the failures, right from within the Test Observability dashboard.

You’d need to configure re-runs and the CI integration (shown in steps below) to be able to achieve the above.

Configure re-run setting

You can go to Settings and enable/disable the Re-run configuration as shown below (it is enabled by default):



Settings for Automatic error analysis

Configure CI Integration for enabling job triggers

Once re-run from dashboard setting is enabled, the next step is to configure a CI integration such that BrowserStack is able to trigger CI jobs on your behalf when you choose to re-run specific tests.

You need to visit the Integrations page and configure one of the supported CI integrations. We currently support Jenkins and Azure Pipeline and steps to configure them have been outlined below:

Jenkins

Follow the steps below to configure your Jenkins integration. Every user in your organisation who intends to use this functionality, will have to do this:

  1. Go to configuring Jenkins integration
  2. You’d need to add the BrowserStack plugin in your Jenkins setup, if not already done. Or, update to the latest version of the plugin if you’re using one.
  3. To do that, from your Jenkins dashboard navigate to Manage Jenkins > Manage Plugins and select the Available tab. Locate the plugin by searching for “browserstack-integration” and thereafter install it. You’ll need v1.2.7 or above.
  4. Add your Jenkins username and pre-generated auth token so that BrowserStack is able to trigger jobs on your behalf.
  5. To generate an Auth Token, you’d need to go to Jenkins UI > Your name on the top right > Configure > API Token > Add new token.
  6. Submit



Configure Jenkins integration

Azure Pipelines

Follow the steps below to configure your Azure Pipelines integration. Every user in your organization who intends to use this functionality, will have to do this:

  1. Go to configuring Azure Pipelines integration
  2. BrowserStack Test Observability would need your Azure DevOps Personal Access Token (PAT) that can be used to trigger builds remotely.
  3. To generate a PAT, you’d need to login to Azure Org > User settings > Personal access tokens > New token > set expiry and give access to Build read & execute. You can read more about generating PAT.
  4. Enter the PAT and submit.



Configure Azure integration

Trigger re-runs

Once you’re done with the CI configuration steps, you can trigger re-runs of any failed tests cases as shown below:

  1. Navigate to the Test listing to see any failed test case.
  2. Hover your mouse pointer over the name of the test to see actions come up on the right hand side (where duration shows up generally)

    Configure Azure integration
  3. Hit the re-run icon to see the action modal show up as shown below:

    Re-run icon Re-run modal

We're sorry to hear that. Please share your feedback so we can do better

Contact our Support team for immediate help while we work on improving our docs.

We're continuously improving our docs. We'd love to know what you liked






Thank you for your valuable feedback

Is this page helping you?

Yes
No

We're sorry to hear that. Please share your feedback so we can do better

Contact our Support team for immediate help while we work on improving our docs.

We're continuously improving our docs. We'd love to know what you liked






Thank you for your valuable feedback!

Talk to an Expert
Download Copy