Test orchestration on BrowserStack Automate
Optimize your Automate tests by efficiently managing their execution, reporting, and rerunning.
Test Orchestration is designed to streamline your team’s testing processes. It enhances test execution by automating tasks such as rerunning failed tests, stopping test suites after critical failures, and prioritizing failed tests. These features reduce manual intervention, optimize resource usage, and improve the reliability of test outcomes, allowing teams to focus on resolving defects faster.
With Test Orchestration, you can accelerate feedback loops, quickly identify and address critical issues, and ensure your development and QA teams maintain high product quality throughout the testing cycle.
The current frameworks supported by BrowserStack Test Orchestration are TestNG and JUnit-5 for Java.
Supported strategies
BrowserStack Automate currently supports the following Test Orchestration strategies:
Test Orchestration is BrowserStack can be triggered using the following capability:
Capabilities | Description | Expected values |
---|---|---|
testOrchestrationOptions |
Enable Test orchestration. |
retryTestsOnFailure - Automatically rerun failed tests. abortBuildOnFailure - Stops the execution of a test suite when a critical failure occurs.runPreviouslyFailedFirst - Prioritize running the previously failed tests in the suite first. |
Key Features
Auto Reruns
Automatically rerun failed tests to identify flaky issues, reduce manual intervention, and improve test reliability. By automatically rerunning failed tests, you can address intermittent issues without manually triggering additional test cycles, ensuring more reliable results.
To enable Auto Reruns, configure the feature in your browserstack.yml
file. You can control the number of reruns by setting a limit on how many times a test should be retried.
Fail Fast
Fail Fast stops the execution of a test suite when a critical failure occurs. This feature helps teams save resources and prioritize fixing high-priority issues by preventing further execution of tests that are likely to fail due to existing critical problems.
You can configure Fail Fast by setting thresholds for the number of allowable failures. Once this threshold is reached, the test suite will automatically abort, allowing the team to focus on resolving the most urgent issues.
Prioritizing Failures
Prioritizing failures involves running previously failed tests first in subsequent test cycles. This strategy is useful when faster feedback is crucial, as it helps teams quickly identify if known issues persist or if new issues have emerged.
By prioritizing fixing failures from the last test run, you can expedite feedback and address high-impact bugs earlier in the development process.
Benefits
- Faster Feedback Loops Running failed tests first helps teams get immediate insights into critical issues, reducing the time taken to detect and fix problems.
- Increased Test Efficiency By focusing on high-priority failures, teams can avoid re-running all tests unnecessarily, saving time and resources.
- Early Bug Detection Prioritizing failures ensures that critical issues are addressed sooner, minimizing the risk of issues progressing further into the development cycle.
- Improved Development Focus Identifying recurring or new failures quickly allows developers to concentrate on resolving issues that have a higher likelihood of impacting the end-user experience.
Prioritizing failures is currently supported only for TestNG, and in master-slave setups, it is supported with Jenkins CI/CD.
The following is a sample master-slave script for a Jenkinsfile
in a Jenkins Pipeline:
BrowserStack will be adding more strategies to the above list. If you want any specific Test Orchestration feature, contact support.
We're sorry to hear that. Please share your feedback so we can do better
Contact our Support team for immediate help while we work on improving our docs.
We're continuously improving our docs. We'd love to know what you liked
We're sorry to hear that. Please share your feedback so we can do better
Contact our Support team for immediate help while we work on improving our docs.
We're continuously improving our docs. We'd love to know what you liked
Thank you for your valuable feedback!