Test orchestration & selection on BrowserStack Automate
Optimize your Automate tests by efficiently managing their execution, reporting, and rerunning.
As test suites grow and release cycles speed up, it’s important to optimise what gets tested and how. Test Selection and Orchestration help optimise your build and achieve the benefits of:
- Faster feedback and faster product releases through targeted and prioritised runs
- Higher QA productivity via reduced debugging of flaky and non-relevant tests.
- Reduced infrastructure usage & cost by skipping less relevant tests. The BrowserStack SDK allows you to configure strategies such as auto retries, fail fast, rerun failures only/first, etc., to optimise your test builds.
Benefits of test orchestration
- Faster feedback loops: Running failed tests first helps teams get immediate insights into critical issues, reducing the time taken to detect and fix problems.
- Increased test efficiency: By focusing on high-priority failures, teams can avoid re-running all tests unnecessarily, saving time and resources.
- Early bug detection: Prioritizing failures ensures that critical issues are addressed sooner, minimizing the risk of issues progressing further into the development cycle.
- Improved development focus: Identifying recurring or new failures quickly allows developers to concentrate on resolving issues that have a higher likelihood of impacting the end-user experience.
The following are supported testing frameworks:
- TestNG and JUnit-5 for Java
- Pytest for Python
- Jest for Node
- Nunit for C#
Using the BrowserStack SDK to integrate with Automate offers significant benefits in stability, performance, and ease of management. Learn how to integrate your tests with BrowserStack SDK.
From the Automate dashboard, you can expand the Test Orchestration menu to see the time saved by each strategy and begin configuration.
Supported strategies
BrowserStack Automate supports the following strategies, which you can enable using the testOrchestrationOptions
capability in your browserstack.yml
file. The strategies are mutually exclusive of each other.
- Auto reruns
- Fail fast
- Run failures only
- Skip flaky and failing
- Prioritizing failures
- Smart Test Selection
Capability | Description |
---|---|
retryTestsOnFailure |
Automatically rerun failed tests. |
abortBuildOnFailure |
Stops the execution of a test suite when a critical failure occurs. |
rerunPreviouslyFailed |
Run only the tests that failed in the previous build. |
skipFlakyandFailed |
Skips tests based on criteria like previous failures or flakiness. |
runPreviouslyFailedFirst |
Prioritize running the previously failed tests in the suite first. |
Clicking Configure opens a modal where you can toggle your preferred strategies and copy the generated browserstack.yml
code.
Auto reruns
Automatically reruns failed tests to identify and manage flaky tests. This strategy helps you confirm whether a failure is persistent or intermittent, improving the reliability of your test results without manual intervention.
To enable auto reruns, configure the feature in your browserstack.yml
file. You can control the number of reruns by setting a limit on how many times a test should be retried.
Fail fast
Fail fast stops the execution of a test suite when a critical failure occurs. This feature helps teams save resources and prioritize fixing high-priority issues by preventing further execution of tests that are likely to fail due to existing critical problems.
You can configure Fail Fast by setting thresholds for the number of allowable failures. Once this threshold is reached, the test suite will automatically abort, allowing the team to focus on resolving the most urgent issues.
Run failures only
Reruns only the tests that failed in the previous build. This provides a rapid feedback loop for developers and QA engineers who need to quickly verify a bug fix without executing the entire test suite.
Although this selection strategy is not yet reflected on the dashboard, you can enable it through the capabilities configuration. This will execute all tests that match the selection criteria, allowing you to benefit from faster feedback.
You can enable this strategy in your browserstack.yml
file.
Skip flaky and failing
Runs a test suite while temporarily ignoring tests that are known to be failing or flaky. This is useful when you want to assess the overall health of an application without being blocked by known issues, such as tests that are failing intermittently or are awaiting a fix.
Although this selection strategy is not yet reflected on the dashboard, you can enable it through the capabilities configuration. This will execute all tests that match the selection criteria, allowing you to benefit from faster feedback.
Prioritizing failures
Prioritizing failures involves running previously failed tests first in subsequent test cycles. This strategy is useful when faster feedback is crucial, as it helps teams quickly identify if known issues persist or if new issues have emerged.
By prioritizing fixing failures from the last test run, you can expedite feedback and address high-impact bugs earlier in the development process.
Once configured, the main build screen displays which orchestration strategies, like Auto Rerun and Prioritizing failures, were active for that specific run.
Prioritizing failures is currently supported only for TestNG, and in master-slave setups, it is supported with Jenkins CI/CD.
Example: Jenkins pipeline integration
To see how Test Orchestration can be applied in a CI/CD environment, you can use the following Jenkinsfile
script. This example configures a parallel pipeline that distributes the test suite across multiple nodes, passing the required BrowserStack environment variables to each job for seamless integration.
The list of supported strategies is actively being expanded. If you have a requirement for a specific Test Orchestration feature that is not listed, we encourage you to submit a feature request by contacting BrowserStack support.
Smart Test Selection
Smart Test Selection uses AI to choose and run a small, prioritized subset of your tests based on recent code changes. This gives you faster feedback on builds and improves test efficiency. Learn how to use BrowserStack’s Smart Test Selection
We're sorry to hear that. Please share your feedback so we can do better
Contact our Support team for immediate help while we work on improving our docs.
We're continuously improving our docs. We'd love to know what you liked
We're sorry to hear that. Please share your feedback so we can do better
Contact our Support team for immediate help while we work on improving our docs.
We're continuously improving our docs. We'd love to know what you liked
Thank you for your valuable feedback!