Fix flaky tests

Fix flaky web tests in Test Companion. Find the tests that pass and fail across runs, then diagnose the cause from their run history and stabilize them.

A flaky test passes on some runs and fails on others, with no change to the code under test. BrowserStack Test Reporting and Analytics marks such a test with the Flaky Smart Tag. Test Companion lists the marked tests in the Failure Analysis panel. It offers a fix for each marked test, even when the latest run of that test passed.

A single failed run does not explain a flaky test. So Test Companion reads the recent failed runs of the test across builds and diagnoses the cause from that pattern. It then changes the test to remove the cause. The fix never skips the test, raises a retry count, or adds a fixed wait.

Prerequisites

You need the following before you start:

• A BrowserStack Test Reporting and Analytics project that receives your build runs.
• One of the following versions of the Test Companion extension:
  • Visual Studio Code: Test Companion 1.31.9 or later
  • JetBrains IDEs: Test Companion 1.8.5 or later
• At least one failed run of the test in the last 10 builds. Test Companion collects the evidence from those failed runs.

Find flaky tests in the Failure Analysis panel

To list the flaky tests in a build, follow these steps:

1. Click the Failure Analysis icon in the Test Companion panel.

   

2. From the project dropdown (see annotation 1), select the project that contains your build.

3. From the Build dropdown (see annotation 2), select the build run you want to work on.

   

4. Click All, Web, or App (see annotation 3) to filter the list by platform.
5. Click Flaky (see annotation 4).

The list now shows only the tests that carry the Flaky Smart Tag in this build. Each row shows a Flaky badge next to the status of the test. A test whose latest run passed still carries the badge. Hover over the badge to read why the test is listed.

Fix one flaky test

In the row of the flaky test, under Actions, click Fix.

After you click Fix, Test Companion does the following:

1. Opens a new task. The test appears as a chip above the chat box, under the label 1 failed test selected.
2. Reads the run history of the test. The progress line under the chip reads Fetching flaky-test history.
3. Fetches the root cause analysis for up to three of the most recent failed runs from the last 10 builds. The progress line reads Fetching root-cause analysis, followed by a count.
4. Fills the chat box with the prompt.

The prompt contains the following details:

• The test name, the suite, and the path.
• The source file, when the source file is known.
• Each recent failed run, with the build number, run ID, failure type, root cause, and suggested fix. A failed run that has no analysis yet is marked RCA not available for this run.

Review the prompt, then press Enter. Test Companion waits for you to send the prompt.

Test Companion reads the failed runs as one pattern, not as separate errors. It first posts the diagnosed cause, the evidence, and the proposed fix in the chat. Then it asks you to confirm. After you confirm, it applies the fix and re-runs the test several times.

The evidence can also point outside the test, to the environment, the infrastructure, or a race in the product. In that case, Test Companion reports that cause with the evidence and does not change the test.

Fix flaky and failed tests together

You do not select any test for this flow. On the All tab, Fix failures covers every failed test and every flaky test in the build.

1. Click All to show every test in the build.

2. Click Fix failures. Test Companion pre-fills a prompt that lists every failed test in the build and includes the flaky tests.

   

3. Review the prompt, then press Enter.

In the prompt, the failed tests are grouped by shared root cause. The flaky tests appear in a separate section of the prompt. That section instructs Test Companion to stabilize each flaky test rather than mask it. Test Companion presents one consolidated plan and waits for your approval before it changes any code.

To fix more than one flaky test as a batch, select them on the Flaky tab. Then click the Fix button that shows your count, for example, Fix 2 Failures. To drop a test from the batch, click the X on the chip for that test.

The latest run of a flaky test may have passed. So Test Companion looks across builds for the most recent failed run of each flaky test in the batch. It then fetches the root cause analysis of that run.

Review and apply the fix

Before modifying any files, Test Companion creates a checkpoint of your workspace. You can use Changes to review the edits or Restore to revert them.

When the task completes, Test Companion posts a summary in the chat. The summary states the diagnosed cause and the change Test Companion made. The chat lists each changed file with a count of added and removed lines.

Review the changes, then choose one of the following actions:

• Click Keep to accept the changes.
• Click Undo to revert them.

A flaky test can pass by chance, so one passing run does not prove that the test is stable. Test Companion re-runs the test several times as part of the task. The Flaky badge in the panel follows the Smart Tag rules in Test Reporting and Analytics. The badge clears only after the test records enough stable runs.

Changes that Test Companion does not make

The prompt forbids the following changes, because each of these changes hides the flakiness instead of removing the cause:

• Skipping the test, or letting the test skip itself.
• Raising the retry count.
• Adding a fixed delay, such as a sleep or a pause.
• Weakening or removing the failing assertion.

If a proposed change falls into one of these categories, ask Test Companion to address the cause instead.

Common causes of flakiness

Test Companion diagnoses and fixes the following causes in web tests:

• Timing and animation: The test acts before an element is visible, before an animation completes, or before async data loads. Test Companion replaces a fixed wait with a wait for the specific condition.
• Network and data synchronization: The test makes an assertion before a request completes. Test Companion waits for the network or data state that the assertion depends on.
• Element actionability: The element is in the DOM but not yet clickable. Test Companion waits for the element to become actionable before it interacts.
• Selector brittleness: A selector matches different elements across runs. Test Companion tightens the selector to one stable target.
• State leakage between tests: A previous test leaves behind data or state that changes the outcome of this test. Test Companion isolates the setup or the teardown.
• Order and shared fixtures: The test passes on its own but fails after another test modifies a shared fixture. Test Companion removes the dependency between the tests.
• Environment or infrastructure: The instability is outside the test. Test Companion reports the instability with the evidence and does not patch the test.

Example scenarios

Each scenario shows what you see, what Test Companion finds in the run history, and what it changes.

Race with an API call

• Symptom: The test passes 70% of the time. The rest of the time, it fails with an element-not-visible error.
• Cause: Across the failed runs, the test clicks a button before an API call completes.
• Fix: Test Companion replaces the fixed wait with a wait for the network to become idle.

Shared fixture changed by another test

• Symptom: The test passes on its own but fails whenever it runs after a particular test in the suite.
• Cause: The earlier test modifies a fixture that both tests share.
• Fix: Test Companion isolates that shared state, so the test no longer depends on run order.

Unstable CI runner

• Symptom: The test fails intermittently, and only on one CI runner, with a connection-reset error.
• Cause: The failed runs all come from that runner, so the environment is unstable, not the test.
• Fix: Test Companion reports the finding in the chat and does not change the test.

Next steps

• Fix failed tests: Fix tests that fail outright, from the panel, the Automate dashboard, a CI run, or the chat.
• Smart Tags in Test Reporting and Analytics: Change the rules that decide when a test is marked as flaky.
• AI settings: Configure auto-approve to let Test Companion apply fixes without asking.