Test Runs

Learn what are Test Runs and how you can launch them either manually or through automation.

A Test Run is a collection of instances of test cases with necessary information like who is handling the involved test cases at a particular time and the state they are in which can be untested, passed, failed, blocked, skipped or to be retested. The results of the test cases in test runs can be updated manually by testers or uploaded through automation.

Automation results can be pushed either by integrating with BrowserStack’s Test Observability or by uploading standard JUnit-XML/BDD-JSON based reports generated by most of the testing frameworks such as TestNG, WebdriverIO, Mocha, Cypress, Playwright, Pytest, Cucumber, Robot, Selenium, REST Assured and more.

Test Runs help you record the results of different tests performed in a testing environment at a particular time or period of time and raise issues or defects(if any). Testing can be performed for software releases, environments like System Integration Testing(SIT)/User Acceptance Testing(UAT), and different browsers or devices.

A user can create a test run and add test cases to be part of the execution and assign them to the owner responsible for the testing. Every test case in a test run has its individual status and can be updated as many times as for an active test run. The status of a test run is the overall progress for the latest status of entire test cases which are part of it.

The status of a test case can only be updated by creating a new instance of a test run or linking it with an existing one.

After the testing is over, a test run can be marked as closed. Note, once a test run is marked as closedm, it cannot be reopened. This is to ensure histroical reports are preserved and testing progress can be tracked for your team.

There are two types of test runs based on the way testing is performed:
- Manual test runs
- Automation test runs

Manual test runs

When a user records the results of the testing performed on a software application in a spreadsheet, document or tool by executing them manually.

How to create manual test runs?

1. Click Test Run on the left navigation panel.
2. Click Create Test Run on the top-right corner.
3. Enter the name of the test run and other basic details.
4. The test run name automatically populates as Test Run - <Date> in the Create Test Run form.
5. Add test cases by selecting from the test cases repository.
6. A test run is in New state by default (implying not started).

How to add results?

1. Open the Test Run which has to be executed.
2. Click Status dropdown of the test case and select the status.
3. Add Status pop-up opens with the selected status.
4. Add or modify Jira issues and other details.
5. Click Add Result to record the status.

Automation test runs

The automation test run is a process of recording the result from the report generated by the execution of the automated test cases by software tools in a repetitive and data-intensive manner. The automation testing is triggered manually or scheduled on events and actions. The linked test case results are updated in the BrowserStack Test Management tool.

Supported tools for automation test runs

Test Management allows integrating automation test run results from options such as Test Observability (via BrowserStack SDK), or JUnit-XML/BDD-JSON based report upload via curl command.

Test Observability

BrowserStack Test Observability currently supports following automation test frameworks:
- TestNG
- WebdriverIO
- Mocha

Once you use Test Observability, your build run information will automatically start flowing in Test Management’s Test Runs section.

Refer to this link for BrowserStack Test Observability integration

JUnit-XML/BDD-JSON based report upload

JUnit-XML/BDD-JSON based report upload in Test Management is supported for following frameworks:
- TestNG
- WebdriverIO
- Playwright
- Cypress
- Robot
- Cucumber
- Mocha
- Behave
- Pytest
- Selenium
- Espresso
- REST Assured

You can do a curl upload of these reports to see results automatically in Test Runs section.

Refer to this link for JUnit-XML/BDD-JSON based report integration