What are Smart Tags

Automatically tag your builds and test runs to help identify issues faster

BrowserStack Test Reporting & Analytics automatically tags your tests with Smart Tags to help you identify issues in your test suite faster. You can customize Smart Tags with custom rules so you can adapt them to suit your needs.

Different types of Smart Tags

Test Reporting & Analytics currently supports the following Smart Tags:

• Flaky
• New Failure
• Always Failing
• Performance Anomaly

All Smart Tags are enabled by default and come with default configurations that should suit most teams. However, you can turn off Smart Tags selectively and customize Smart Tag configurations in the Settings.

Smart Tags are applied at the test case level. However, they are also shown at the build level for builds with test cases that have Smart Tags applied.

Flaky

Test Reporting & Analytics helps you identify tests that are flaky, and automatically Smart Tags them as “Flaky”. Any builds with flaky tests that meet the flakiness rules will also be tagged.

Conditions for the Flaky Smart Tag to be applied

Tests will be marked Flaky if either of the two conditions below are met:

1. By default, if the test passes on a retry attempt in the same run across the last 10 consecutive runs. You may customize this and set a value anywhere between 0 to 30. Setting this to 0 means a test would be marked as flaky only if it passes a retry in the latest run.
2. By default, if the test status flips from pass to fail (or vice versa) more than 50% in the last 10 runs. You can customize both the % value and the last X runs parameters. The maximum value you can set for X is 30.

New Failure

Test Reporting & Analytics automatically identifies brand new test failures previously unseen or recently unseen so you can quickly understand if something new has broken in your application.

Conditions for the New Failure Smart Tag to be applied

Tests will be marked New Failure if the following condition is met:

• By default, any test that fails with a new error for the first time in the last 5 runs.
• However, you can reconfigure this Smart Tag to be applied for any test failing with any error or new error in the last X runs.

Always Failing

Test Reporting & Analytics automatically identifies test cases that have been continuously failing and need repair.

Conditions for the Always Failing Smart Tag to be applied

Tests will be marked Always Failing if the following condition is met:

• By default, any test failing with the same error in the last 5 runs.
• However, you can reconfigure this Smart Tag to be applied for any test failing with any error or the same error in the last X runs. You can set the value of X between 2 and 30.

Performance Anomaly

Test Reporting & Analytics automatically identifies test runs that took much longer to execute and could be contributing to performance issues in your suite.

Conditions for the Performance Anomaly Smart Tag to be applied

Tests will be marked Performance Anomaly if the following condition is met:

• By default, any test with a run duration above the 95th percentile among the last 100 executions.
• However, you can reconfigure this Smart Tag and customize both the percentile value (between 80th and 99th percentile) and the number of executions (between 30 and 100 executions).

Test Reporting & Analytics takes into account a maximum of 100 days of data for this calculation.

How to customize Smart Tags

You can selectively turn Smart Tags on or off, and customize Smart Tag configurations. Configurations are maintained at a project level. For each unique project you have on Test Reporting & Analytics, you can configure and maintain separate settings.

Follow these steps to customize Smart Tags:

1. Go to the Settings menu.
2. Select Enable against the Smart Tags which you want to turn on, and Disable against those you want to turn off.
3. In each Smart Tag, you can further select and deselct options, and configure the rule using the menus provided.
4. Click Save changes.