Find pull requests that broke a build

Find which merged pull request broke a failed GitHub Actions or Jenkins build, and which ones are safe to ship, from the Test Companion chat, panel, or CLI.

A CI build fails before a release. You need to know which change caused it, and whether the other changes can still ship. Test Companion compares the failed build with the previous build of the same workflow. It excludes tests that were already failing. Then it traces each new failure to the pull request that caused it. Test Companion answers with one verdict per pull request.

Test Companion only reads the build data and the pull requests. It does not comment on a pull request, set a status check, revert a change, or post anywhere.

Prerequisites

You need the following before you start:

• CI failure analysis enabled for your BrowserStack account. To enable this feature, contact BrowserStack support.
• A failed build in GitHub Actions or Jenkins for a repository hosted on GitHub.com.
• The GitHub CLI, gh, installed and signed in. Test Companion reads the commits and the pull request diffs with gh. See Connect GitHub Actions to Test Companion.
• For a Jenkins build, a Jenkins API token and a job that records the git commit it built. See Connect Jenkins to Test Companion.
• To start from the Failure Analysis panel, a Test Reporting and Analytics project that receives the build. Your BrowserStack credentials must be configured in Test Companion.
• A supported IDE or the CLI. See Availability.

How it works

Test Companion follows these steps:

1. Finds the baseline build. In GitHub Actions, that is the previous run of the same workflow on the same branch. In Jenkins, that is the nearest earlier build of the job that finished.
2. Excludes failures that already existed in the baseline. Only new failures count against the changes in this build.
3. Groups the new failures by their error message.
4. Lists the pull requests merged between the baseline and this build, and reads the diff of each one.
5. Traces each failure group to the change that caused it, and gives every pull request a verdict.

Find bad PRs from the chat

Paste the link to the failed CI run and ask which pull request introduced the failure. For example:

Which PR broke this build? https://github.com/<org>/<repo>/actions/runs/<run-id>

For a Jenkins build, paste the build URL instead:

Which changes are safe to ship? https://<jenkins-host>/job/<job-name>/<build-number>/

Test Companion starts the analysis when your message contains the run link and asks about one of the following:

• The pull request, change, or commit that broke the build.
• The changes that are safe to ship.
• The pull requests that are bad PRs.

Test Companion fetches the failure evidence from the run, finds the baseline, and reads the pull request diffs. It reports each step in the chat while it works.

Find bad PRs from the Failure Analysis panel

To start the analysis from a build in Test Reporting and Analytics, follow these steps:

1. Click the Failure Analysis icon in the Test Companion panel.
2. From the project dropdown, select the project that contains your build.
3. From the Build dropdown, select a build with at least one failed test.
4. Without selecting a test, click the arrow next to Fix failures, and then click Find bad PRs.

Test Companion opens a new task and fills the chat box. It sends nothing until you press Enter. The text it fills in depends on your Test Reporting and Analytics project:

• Release readiness is turned on: Test Companion fills the chat box with a verdict table. The table comes from the release readiness assessment of the project in Test Reporting and Analytics. Test Companion does not analyze any diff for this table, so no row cites a diff line.
• Release readiness is turned off: Test Companion fills the chat box with a prompt. The prompt has a placeholder for the CI run link. Replace the placeholder with the link to the CI run of this build, then press Enter.

Find bad PRs from the CLI

Run tc verdict with the URL of the failed run:

tc verdict --run https://github.com/<org>/<repo>/actions/runs/<run-id>

The command accepts either a GitHub Actions run URL or a Jenkins build URL. It does not accept a build number. Add --json for machine-readable output.

The verdict workflow is also available to tc workflow run in a pipeline. See Run the CLI in a CI/CD pipeline.

Read the verdicts

Test Companion answers in the chat with a table. Each row is one pull request, with its verdict, the reason, the evidence, and the next action.

The following table lists the five verdicts:

A pull request gets This one broke it only when its diff contains the line that a failure traces to. Test Companion quotes that line. A change that only touches the code the failing tests run through gets Validate first.

Below the table, Test Companion lists what it did not analyze under Not analysed. It ends with a summary of the build.

Limits

Test Companion analyzes only part of the build in the following cases:

• More than 10 pull requests: Test Companion analyzes the 10 most recent pull requests merged since the baseline. It lists the rest under Not analysed.
• Direct pushes: A commit that landed without a pull request is listed, but it gets no verdict.
• No baseline: When no earlier build has a usable result, new failures cannot be separated from old ones. For GitHub Actions, no pull request is cleared. For Jenkins, every failure is treated as new.
• Large diffs: Test Companion reads only the start of a very large diff. The answer marks each diff that was cut short.

Troubleshooting

Use the following fixes for common issues:

• Find bad PRs does not appear in the panel: The selected build has no failed tests. The action appears only for a build with at least one failed test.
• Test Companion says that CI failure analysis is not enabled for your account: Contact BrowserStack support to get access.
• Test Companion asks for the GitHub CLI: Install gh from cli.github.com. Run gh auth login, then send the run link again.
• Test Companion could not fetch the run: Run gh auth status to confirm that your GitHub account can access the repository. Then check that the run still exists, because GitHub keeps run logs for a limited time.
• No verdicts for a Jenkins build: The job does not record the commit that it built. Use the Jenkins Git plugin with a GitHub remote. The repository must be on GitHub.
• You got a root cause analysis instead: Your message did not ask which change broke the build. A message that asks for a fix also starts a fix instead. Ask, for example, Which PR broke this build? with the run link.

Availability

Next steps

• Fix failed tests: Fix the failures from the panel, a CI run, or the chat.
• Fix flaky tests: Find tests that pass and fail across runs, and stabilize them.
• Connect GitHub Actions to Test Companion: Set up the GitHub CLI and the GitHub Actions connection that this analysis uses.