Smart Test Selection Agent

Learn how to use BrowserStack’s Smart Test Selection to run tests relevant to your code changes, get faster feedback, and increase test efficiency.

What is Smart Test Selection?

Smart Test Selection uses AI to identify and run tests impacted by recent code changes. Instead of executing the entire test suite after every change, you can run a relevant subset of tests to get faster feedback on your builds

How Smart Test Selection Works

The Smart Test Selection agent analyzes the relationship between the metadata of your code changes, such as file names, file paths, commit authors, etc. and historical test execution results to identify which tests are most likely to fail.

You can run tests selected by the AI agent in one of the following modes:

• relevantFirst : Runs the selected tests first, followed by the remaining suite.
  This mode ensures faster feedback on critical issues with full coverage
• relevantOnly: Runs only the selected tests and skips the rest. This mode maximizes efficiency and reduces execution time, infrastructure costs

Learning Mode

Once enabled, the AI agent enters a learning mode. It learns how your code changes affect your tests by analysing your project’s build and commit history. The agent does not select any tests during this period. To help the AI learn faster, you can ensure the following:

• Frequent builds: Run builds several times a week
• Diverse commits: Provide a rich history of changes across the codebase
• Test failures: A suite with a consistent failure rate (~10%)

Selecting Tests

After achieving target accuracy in the learning mode, the AI agent starts selecting tests for each new build. It only selects tests when it has high confidence in its predictions.

• Sufficient overlap: The AI agent selects and runs relevant tests based on your chosen mode (relevantFirst or relevantOnly)
• Insufficient overlap: The AI agent skips selection for the current build and continues learning from the data

Enable Smart Test Selection

Prerequisite

Ensure you have activated BrowserStack AI in your organization’s settings. For more details, see activate BrowserStack AI preferences.

Follow these steps to enable Smart Test Selection in your project:

{
  "UTILS_SERVICE": {
    "url": "https://github.com/org/utils-service",
    "baseBranch": "main"
    // optional. Uses default branch from Git if not specified
  },
  "API": {
    "url": "https://github.com/org/api",
    "baseBranch": "main"
    // optional. Uses default branch from Git if not specified
  }
}

testOrchestrationOptions:
  runSmartSelection:
    enabled: true
    mode: 'relevantFirst' # or 'relevantOnly'
    source: './selection_config.json'

export BROWSERSTACK_ORCHESTRATION_SMART_SELECTION_FEATURE_BRANCHES='{"UTILS_SERVICE":"feature-A","API":"hotfix-B"}'

View Results

Once the AI agent starts selecting tests, you can monitor its impact on your dashboard. The Build Optimisation Widget shows the percentage of tests skipped and the total time saved.

Frequently Asked Questions (FAQ)

1. Can I configure a list of “must-run” tests that are never skipped?

Not at this time. If you have a set of critical tests that must always be run, we strongly recommend using the relevantFirst mode. This ensures your critical tests are run as part of the full suite while still benefiting from prioritized feedback. The ability to configure a specific “must-run” list is planned for a future update.

2. How do I know when the “learning mode” is complete?

Currently, the transition from learning to active mode is automatic. Once the AI is active, you will begin to see the data related to time saved and percentage of tests dropped appearing in the Build Optimisation widget on your dashboard.

If you are facing any issues using the feature, contact us.