Fix flaky app tests

Fix flaky app tests in Test Companion. Find the tests that pass and fail across runs, then diagnose the cause from their run history and stabilize them.

A flaky app test passes on some runs and fails on others. Neither the app nor the test changed in between. BrowserStack Test Reporting and Analytics marks such a test with the Flaky Smart Tag. Test Companion lists the marked tests in the Failure Analysis panel. It offers a fix for each marked test, even when the latest run of that test passed.

A single failed run does not explain a flaky test. So Test Companion reads the recent failed runs of the test across builds and diagnoses the cause from that pattern. It then changes the test to remove the cause. The fix never skips the test, raises a retry count, or adds a fixed wait.

Prerequisites

You need the following before you start:

• A BrowserStack Test Reporting and Analytics project that receives your App Automate build runs.
• One of the following versions of the Test Companion extension:
  • Visual Studio Code: Test Companion 1.31.9 or later
  • JetBrains IDEs: Test Companion 1.8.5 or later
• At least one failed run of the test in the last 10 builds. Test Companion collects the evidence from those failed runs.

Find flaky app tests in the Failure Analysis panel

To list the flaky app tests in a build, follow these steps:

1. Click the Failure Analysis icon in the Test Companion panel.

   

2. From the project dropdown (see annotation 1), select the project that contains your app build.

3. From the Build dropdown (see annotation 2), select the build run you want to work on.

   

4. Click App to show only the tests that ran on a device.

5. Click Flaky.

   

The list now shows only the app tests that carry the Flaky Smart Tag in this build. Each row shows a Flaky badge next to the status of the test. A test whose latest run passed still carries the badge. Hover over the badge to read why the test is listed.

Fix one flaky app test

In the row of the flaky test, under Actions, click Fix.

After you click Fix, Test Companion does the following:

1. Opens a new task. The test appears as a chip above the chat box, under the label 1 failed test selected.
2. Reads the run history of the test. The progress line under the chip reads Fetching flaky-test history.
3. Fetches the root cause analysis for up to three of the most recent failed runs from the last 10 builds. The progress line reads Fetching root-cause analysis, followed by a count.
4. Fills the chat box with the prompt.

The prompt contains the following details:

• The test name, the suite, and the path.
• The source file, when the source file is known.
• The device and the OS that the run used.
• Each recent failed run, with the build number, run ID, failure type, root cause, and suggested fix. A failed run that has no analysis yet is marked RCA not available for this run.

Review the prompt, then press Enter. Test Companion waits for you to send the prompt.

Test Companion first posts the diagnosed cause, the evidence, and the proposed fix in the chat. Then it asks you to confirm. After you confirm, it applies the fix and re-runs the test several times.

The evidence can also point outside the test, to the device, the infrastructure, or a race in the app. In that case, Test Companion reports that cause with the evidence and does not change the test.

Fix flaky and failed app tests together

You do not select any test for this flow. On the All tab, Fix failures covers every failed test and every flaky test in the build.

1. Click All to show every test in the build.

2. Click Fix failures. Test Companion pre-fills a prompt that lists every failed test in the build and includes the flaky tests.

   

3. Review the prompt, then press Enter.

In the prompt, the failed tests are grouped by shared root cause. The flaky tests appear in a separate section of the prompt. That section instructs Test Companion to stabilize each flaky test rather than mask it. Test Companion presents one consolidated plan and waits for your approval before it changes any code.

To fix more than one flaky test as a batch, select them on the Flaky tab. Then click the Fix button that shows your count, for example, Fix 2 Failures. To drop a test from the batch, click the X on the chip for that test.

The latest run of a flaky test may have passed. So Test Companion looks across builds for the most recent failed run of each flaky test in the batch. It then fetches the root cause analysis of that run.

Review and apply the fix

Before modifying any files, Test Companion creates a checkpoint of your workspace. You can use Changes to review the edits or Restore to revert them.

When the task completes, Test Companion posts a summary in the chat. The summary states the diagnosed cause and the change Test Companion made. The chat lists each changed file with a count of added and removed lines.

Review the changes, then choose one of the following actions:

• Click Keep to accept the changes.
• Click Undo to revert them.

Re-run the test on the same device and OS version that produced the failures. A pass on a different device does not prove that the original flakiness is gone. The Flaky badge in the panel follows the Smart Tag rules in Test Reporting and Analytics. The badge clears only after the test records enough stable runs.

Changes that Test Companion does not make

The prompt forbids the following changes, because each of these changes hides the flakiness instead of removing the cause:

• Skipping the test, or letting the test skip itself.
• Raising the retry count.
• Adding a fixed delay, such as a sleep or a pause.
• Weakening or removing the failing assertion.

If a proposed change falls into one of these categories, ask Test Companion to address the cause instead.

Common causes of flakiness in app tests

Test Companion diagnoses and fixes the following causes in app tests:

• Animation and transition timing: The test taps before a screen transition or an animation completes. Test Companion replaces a fixed wait with a wait for the element to reach the expected state.
• Network and data synchronization: The test makes an assertion before an on-device request completes. Test Companion waits for the state that the assertion depends on.
• Element actionability: The element is in the UI hierarchy but not yet tappable. Test Companion waits for the element to become actionable before it interacts.
• Locator drift: An accessibility ID or a resource ID resolves to different elements across runs. Test Companion tightens the locator to one stable target.
• Leftover device state: A permission, a cached file, or a signed-in session from an earlier run changes the test outcome. Test Companion resets that state in the setup or the teardown.
• Platform divergence: The test is stable on one OS and flaky on the other. Test Companion changes only the code branch for the affected platform.
• Environment or infrastructure: The instability is outside the test. Test Companion reports the instability with the evidence and does not patch the test.

Example scenarios

Each scenario shows what you see, what Test Companion finds in the run history, and what it changes.

Tap before an animation finishes

• Symptom: A test that taps through a slide-in tutorial passes 70% of the time.
• Cause: In the failed runs, the tap fires before the animation finishes.
• Fix: Test Companion replaces the implicit wait with a wait for the animation to finish.

Permission left over from an earlier test

• Symptom: A camera-scan test passes when the run starts clean but fails after an earlier test used the camera permission.
• Cause: The earlier test granted or denied the permission, and that state carries over.
• Fix: Test Companion resets the permission in the setup, so the test starts from a known state.

Keyboard covers a button on one platform

• Symptom: A signup test is stable on an iPhone but flaky on a Pixel.
• Cause: On the Pixel, the soft keyboard sometimes covers the Submit button.
• Fix: Test Companion adds a step that scrolls to the button before the tap, on the Android branch only.

Next steps

• Fix failed app tests: Fix app tests that fail outright, from the panel or the chat.
• Smart Tags in Test Reporting and Analytics: Change the rules that decide when a test is marked as flaky.
• AI settings: Configure auto-approve to let Test Companion apply fixes without asking.