Build status calculation and visibility of tests and hooks

Learn how build status is calculated in Test Reporting & Analytics.

Build status

Build status in Test Reporting & Analytics helps you quickly identify how your builds and the associated tests ran.

Test Reporting & Analytics marks every build into one of the following statuses based on the status of the tests in the build:

• Passed
• Failed
• Skipped
• Running
• Unknown

When a build starts, Test Reporting & Analytics receives a build start event and temporarily marks the build status of the build as running. When the build runs, it receives multiple events from the testing framework one after the other. Test Reporting & Analytics continues to listen to all these events until it receives the build finish event or when an Inactivity timeout occurs.

The following table details how Test Reporting & Analytics determines the status of a build:

Inactivity timeout

When you run a build, Test Reporting & Analytics receives multiple events from your test executions.

If the waiting time between receiving two events breaches a predefined limit, the build times out due to inactivity. This could happen due to issues like intermittent network connectivity issues, abrupt CI job crashes, and several other reasons. When inactivity timeout occurs, if Test Reporting & Analytics has not received the test finish event of any test, it marks the status of such tests as unknown.

Modify the inactivity timeout limit

By default, the inactivity timeout limit is 300 seconds. You can change this predefined limit as per your requirements.

Follow these steps to modify the inactivity timeout limit in Test Reporting & Analytics:

1. Go to Settings > General.

2. Update Timeout for build and click Save changes.

You can contact support if you continue to see that tests are marked as unknown even after modifying the inactivity timeout limit.

Visibility of hooks and tests

Test count in Test Reporting & Analytics

Test Reporting & Analytics tracks the total number of tests in a build, along with counts of passed, failed, skipped, and unknown tests.

You can view the total test count in the Build Summary widget in Build Insights. You can also view the number of passed, failed, skipped, and unknown tests at the top-right-hand corner of the Build Insights page and also in a few other widgets.

The following screenshot highlights the test count and the test status count displayed in Build Insights:

How hooks are handled in Test Reporting & Analytics

Test Reporting & Analytics handles hooks that run with each test case (like beforeEach and afterEach) differently from the hooks that run once for a block of test cases or at a file level (global hooks like beforeAll, and afterAll).

Hooks that run with each test case are always excluded from the test listing by Test Reporting & Analytics. You can view such hooks only inside a test.

In contrast, Test Reporting & Analytics provides you with four options to handle the hooks that run once for a block of tests or at a file level (global hooks):

• Do not show any global hooks in the test listing.
• Show only failed beforeAll hooks in the test listing. This is the default setting.
• Show only failed hooks in the test listing so that you get visibility on all types of failures.
• Show all hooks (both passing and failing) in the test listing for complete visibility.

Change hooks visibility in the test count and metrics

For hooks that run once for a block of tests or at a file level (global hooks), Test Reporting & Analytics shows only failed beforeAll hooks in the test listing by default.

However, you can change this setting. Changing it alters the behavior of some features and how Test Reporting & Analytics calculates certain metrics, including test count for billing.

The following main features and metrics change when you modify the global hooks setting:

• Test Listing shows the type of global hooks you selected, as separate tests. If you choose Ignore all hooks, global hooks will not appear in the test listing.
• Aggregate test count and the widgets in Build Insights will include global hooks as separate tests. If you choose Ignore all hooks, the aggregate test count will not include any global hooks.
• Tests Health metrics like Failure count and Failure rate will include such global hooks as separate tests. If you choose Ignore all hooks, these metrics will not include any global hooks.
• Testing Trends metrics and widgets will include global hooks as separate tests. If you choose Ignore all hooks, these metrics and widgets will not include any global hooks.

To select how to treat hooks in your test suite:

1. Go to Settings > General.

2. Under Hooks visibility, select one among Ignore all hooks, Show failed beforeAll hooks only, Show failed hooks only, or Include all hooks.

3. Click Save changes.

Handling hooks within custom alerts

Test Reporting & Analytics allows users to set up custom alerts for every build. You have the option to ignore all hooks, show failed beforeAll hooks only, show failed hooks only, or include all hooks in the calculation of some of these alerts. Which hooks you want to include in the calculation influences the trigger point of these alerts.