Organize test runs in Test Observability

Learn how to organize your test runs into projects and builds to derive the maximum value out of the product.

Organizing structures

In Test Observability, the following are the organizing structures that are available:

• Project
• Build name (a.k.a. your CI job name e.g. Nightly Regression)
• Build number (a.k.a. the sequential job run identifier that is used in conjuction with job name to uniquely identify a run)

How to use Projects in Test Observability?

Test Observability organizes all test executions into Projects and different runs within such Projects. The following pointers underline how to use the Project hierarchy in Test Observability:

1. Project is used as a namespace. Once you’re inside a project, you do not get to see any runs from a different project nor do you get to see any analysis or insights about the tests or runs from a different project.
2. You should use projectName key in the browserstack.yml config file of the SDK and specify a static name for your project that you won’t change for all the runs of the same job.
3. We recommend creating unique projects for each of your pipelines or applications under test.
4. If you have different CI jobs for e.g. Smoke, Sanity, P1, Regression that run a subset of the same test suite every time, then each of those jobs should ideally be part of the same project since you’ll get to analyse the stability and performance of all those test cases together across different type of runs.

How to use build names within projects in Test Observability?

Every time you invoke a CI job, the build name should remain the same. We have a logic of automatically identifying different runs of the same job and hence you need not pass a different buildName every time.

This is how you should configure buildName key in your browserstack.yml:

1. Lets assume your master test suite has 2000 test cases and you run the entire suite across multiple browsers in your nightly regression job. You should then specify a static name for buildName whenever you run the same nightly regression job for e.g. buildName: "Nightly Regression"
2. Now lets say, you run a subset of those 2000 tests (lets assume 400 tests) in your Sanity job that run multiple times in the day whenever a new PR or commit is pushed. In that case, you should pass a different static name in buildName for e.g. “PR Sanity” and thereafter you should use buildIdentifier to identify different PRs or commits (explained in next section).
3. In no scenario, you should use dynamic build names. Because, Test Observability maps historical runs of a test case in the context of the build name. If you pass dynamic build names, you will not be able to see the historical run stats of a test case when you’re viewing the report or while debugging.
4. We have a automatic derived name mapping logic in place which tries to intelligently identify if you’re passing dynamic build names and tries to extract the static part from it. But the logic is never 100% accurate and hence it is always best to adhere to the above guidelines of build naming to derive the maximum value out of the product.

The following is an example of how your report would look if you pass dynamic build names. Note the No historical data found section:

Now, note the following screenshot of the report with test history getting populated (green and red dots on the right) simply because a static build name was used every time:

How to uniquely identify build runs within each static build name?

Test Observability automatically increments the build run number whenever it identifies a different run of the same build name. Hence you see the auto-incrementing numbers against your build name e.g. #14.

You can also do the following to pass on additional information about a specific run:

• Use buildIdentifier key in browserstack.yml and specify any string as a value.
• Test Observability offers a re-run feature wherein you can trigger a re-run job and results of the re-run will update the results of the original run.