Organize test runs in Test Observability

Learn how to organize your test runs into projects and builds to derive the maximum value out of the product.

Organizing structures

In Test Observability, the following are the organizing structures that are available:

• Project
• Build name (a.k.a. your CI job name e.g. Nightly Regression)
• Build number (a.k.a. the sequential job run identifier that is used in conjunction with the job name to uniquely identify a run)

How to use Projects in Test Observability?

Test Observability uses a project hierarchy to organize all test executions into Projects and different runs within those Projects. Here are some important points to keep in mind when using the project hierarchy:

1. Project is used as a namespace, meaning that when you’re inside a project, you only see analysis and runs related to that project.
2. You should specify a static name for your project using the projectName key in the browserstack.yml config file of the SDK.
3. Restrict the characters in your projectName to alphanumeric characters (A-Z, a-z, 0-9), underscores (_), colons (:), and hyphens (-). Any other character will be replaced with a space.
4. It is recommended to create unique projects for each of your pipelines or applications under test.
5. If you have different CI jobs that run a subset of the same test suite every time, it’s best to group them in the same project. This allows you to analyze the stability and performance of all test cases across different types of runs.

You can switch between projects and view the current project using the dropdown on the top-left corner.

How to use build names within projects in Test Observability?

Every time you invoke a CI job, the build name should remain the same. Test Observability automatically identifies different runs of the same job and hence you need not pass a different buildName every time.

Follow these guidelines to configure buildName in your browserstack.yml file to get the most value out of Test Observability:

• Use a static name for buildName when running the same test suite across multiple browsers in your nightly regression job.
• Use a different static name for buildName when running a subset of tests in your sanity job, and use buildIdentifier to identify different PRs or commits.
• Do not use dynamic build names as Test Observability maps historical runs of a test case in the context of the build name, and dynamic build names can lead to incomplete or inaccurate historical run stats.
• Limit the characters in your buildName to alphabets, digits, _, :, and -. Any other character will be replaced by a space.

What happens if you pass dynamic build names every day?

Note the No historical data found section:

How does the history mapping help in visualizing your test reports?

In the following screenshot of the report, you can find the test history populated (green and red dots on the right) as a static build name was used every time:

Different ways of passing build names

Test Observability accepts build names in the following order of priority:

1. Use environment variable BROWSERSTACK_BUILD_NAME.
2. Use the buildName key inside the testObservabilityOptions object in the config file for WebdriverIO. (applicable only for this framework).
3. Use the buildName key in the browserstack.yml file for any framework’s SDK or inside the capabilities object if using WebdriverIO.
4. If none of the above is available, use the folder name of the folder where the config file is found as the build name.

What happens if Test Observability detects dynamic build names?

If you pass dynamic build names to Test Observability, the value of the product’s history timeline offering is diminished. To address this issue, Test Observability has a heuristic-based logic that derives the static part of a build name.

The logic involves:

• Tokenizing the build name into distinct words separated by common separators
• Discarding all alpha-numeric substrings, numbers, and date or timestamps from the name.

After applying the logic, you can see the automatically derived static name in all build name filter dropdowns across the product. However, on the Build Runs page, you can still see the name that you have passed, and you can still search using the name that you’re passing.

The builds where automatic heuristics are applied look like this:

How to uniquely identify build runs within each static build name?

Test Observability automatically increments the build run number whenever it identifies a different run of the same build name. Hence you see the auto-incrementing numbers against your build name e.g. #14.

You can also do the following to pass additional information about a specific run:

• Use the buildIdentifier key in browserstack.yml and specify any string as a value.
• Test Observability offers a re-run feature wherein you can trigger a re-run job and the results of the re-run will update the results of the original run.