Test orchestration & selection on BrowserStack Automate

Optimize your Automate tests by efficiently managing their execution, reporting, and rerunning.

As test suites grow and release cycles speed up, it’s important to optimise what gets tested and how. Test Selection and Orchestration help optimise your build and achieve the benefits of:

• Faster feedback and faster product releases through targeted and prioritised runs
• Higher QA productivity via reduced debugging of flaky and non-relevant tests.
• Reduced infrastructure usage & cost by skipping less relevant tests. The BrowserStack SDK allows you to configure strategies such as auto retries, fail fast, rerun failures only/first, etc., to optimise your test builds.

Benefits of test orchestration

• Faster feedback loops: Running failed tests first helps teams get immediate insights into critical issues, reducing the time taken to detect and fix problems.
• Increased test efficiency: By focusing on high-priority failures, teams can avoid re-running all tests unnecessarily, saving time and resources.
• Early bug detection: Prioritizing failures ensures that critical issues are addressed sooner, minimizing the risk of issues progressing further into the development cycle.
• Improved development focus: Identifying recurring or new failures quickly allows developers to concentrate on resolving issues that have a higher likelihood of impacting the end-user experience.

From the Automate dashboard, you can expand the Test Orchestration menu to see the time saved by each strategy and begin configuration.

Supported strategies

BrowserStack Automate supports the following strategies, which you can enable using the testOrchestrationOptions capability in your browserstack.yml file. The strategies are mutually exclusive of each other.

• Auto reruns
• Fail fast
• Run failures only
• Skip flaky and failing
• Prioritizing failures
• Smart Test Selection

Clicking Configure opens a modal where you can toggle your preferred strategies and copy the generated browserstack.yml code.

Auto reruns

Automatically reruns failed tests to identify and manage flaky tests. This strategy helps you confirm whether a failure is persistent or intermittent, improving the reliability of your test results without manual intervention.

To enable auto reruns, configure the feature in your browserstack.yml file. You can control the number of reruns by setting a limit on how many times a test should be retried.

Fail fast

Fail fast stops the execution of a test suite when a critical failure occurs. This feature helps teams save resources and prioritize fixing high-priority issues by preventing further execution of tests that are likely to fail due to existing critical problems.

You can configure Fail Fast by setting thresholds for the number of allowable failures. Once this threshold is reached, the test suite will automatically abort, allowing the team to focus on resolving the most urgent issues.

Run failures only

Reruns only the tests that failed in the previous build. This provides a rapid feedback loop for developers and QA engineers who need to quickly verify a bug fix without executing the entire test suite.

You can enable this strategy in your browserstack.yml file.

Skip flaky and failing

Runs a test suite while temporarily ignoring tests that are known to be failing or flaky. This is useful when you want to assess the overall health of an application without being blocked by known issues, such as tests that are failing intermittently or are awaiting a fix.

Prioritizing failures

Prioritizing failures involves running previously failed tests first in subsequent test cycles. This strategy is useful when faster feedback is crucial, as it helps teams quickly identify if known issues persist or if new issues have emerged.

By prioritizing fixing failures from the last test run, you can expedite feedback and address high-impact bugs earlier in the development process.

Once configured, the main build screen displays which orchestration strategies, like Auto Rerun and Prioritizing failures, were active for that specific run.

Example: Jenkins pipeline integration

To see how Test Orchestration can be applied in a CI/CD environment, you can use the following Jenkinsfile script. This example configures a parallel pipeline that distributes the test suite across multiple nodes, passing the required BrowserStack environment variables to each job for seamless integration.

The list of supported strategies is actively being expanded. If you have a requirement for a specific Test Orchestration feature that is not listed, we encourage you to submit a feature request by contacting BrowserStack support.

Smart Test Selection

Smart Test Selection uses AI to choose and run a small, prioritized subset of your tests based on recent code changes. This gives you faster feedback on builds and improves test efficiency. Learn how to use BrowserStack’s Smart Test Selection.