Use Test Sharding

Parallel testing enables you to reduce the overall test execution times, resulting in improved build times and faster releases. Test Sharding enables you to test at scale on BrowserStack.

Introduction

By default, all test cases in your Espresso test-suite are executed sequentially. Instead of running your test-suite sequentially, you can split test cases into different groups called shards. Each shard can then be executed in parallel on one or more devices. This is especially helpful when your test-suite has a large number of test cases.

For example, if your test-suite has 500 test cases. Assume each test case takes about a minute to execute. If the test-suite is executed sequentially, the overall execution time will be close to 500 mins. However, if you split your test-suite into 10 shards, each shard will consist of 50 test cases. If each shard is executed in parallel, the total test execution time will be down to 50 mins.

To enable test sharding, you need to configure the shards parameter in the REST API request to start Espresso test execution.

REST API endpoint :

POST /app-automate/espresso/v2/build

Example cURL request:

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
-X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
-d '{"shards": {"numberOfShards": 2}, "devices": ["Google Pixel 3-9.0"], "app": "bs://f7c874f21852ba57957a3fdc33f47514288c4ba4", "testSuite": "bs://j3c874f21852ba57957a3fdc33f47514288c4ba4"}' \
-H "Content-Type: application/json"

In this example, the test-suite will be split into 2 shards, and each of those shards will execute in parallel on Google Pixel 3-9.0. This will block 2 parallel tests licenses in your App Automate subscription plan.

Sharding strategy

There are multiple strategies to control how you want to split the test-suite into different shards.

1. Use auto strategy

This is simplest way to use test sharding. The test-suite is automatically split into the desired number of shards. Each shard will have approximately same number of test cases. We rely on the AndroidJUnitRunner shard feature to implement this behaviour. All the shards will be executed in parallel on all the devices specified in the devices parameter.

REST API endpoint :

POST /app-automate/espresso/v2/build

Example cURL request :

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
-X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
-d '{"shards": {"numberOfShards": 2}, "devices": ["Samsung Galaxy S20-10.0","Google Pixel 3-10.0"], "app": "bs://f7c874f21852ba57957a3fdc33f47514288c4ba4", "testSuite": "bs://f7c874f21852ba57957a3fdc33f47514288c4ba4"}' \
-H "Content-Type: application/json"

In the above example, the test cases will be automatically split into 2 groups (i.e. shards). Both the shards will execute in parallel on Samsung Galaxy S20-10.0 and Google Pixel 3-10.0. This will block 4 parallel tests licenses in your App Automate subscription plan.

2. Use package strategy

Using this strategy, you can group test cases by their fully qualified Java package names (e.g. com.foo.login) into different shards. All the shards will be executed in parallel on the devices specified in the devices parameter.

REST API endpoint :

POST /app-automate/espresso/v2/build

Example cURL request :

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
-X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
-d '{"shards": {"numberOfShards": 2, "mapping": [{"name": "Shard 1", "strategy": "package", "values": ["com.foo.login", "com.foo.logout"]}, {"name": "Shard 2", "strategy": "package", "values": ["com.foo.dashboard"]}]}, "devices": ["Samsung Galaxy S20-10.0","Google Pixel 3-10.0"], "app": "bs://f7c874f21852ba57957a3fdc33f47514288c4ba4", "testSuite": "bs://e994db8333e32a5863938666c3c3491e778352ff"}' \
-H "Content-Type: application/json"

In the above example, the test cases will be split into 2 groups (i.e. shards). The first shard will consist of test cases belonging to com.foo.login and com.foo.logout packages. The second shard will consist of test cases belong to the com.foo.dashboard package. Both the shards will execute in parallel on Samsung Galaxy S20-10.0 and Google Pixel 3-10.0. This will block 4 parallel tests licenses in your App Automate subscription plan.

3. Use class strategy

Using this strategy, you can group test cases by their fully qualified Java class names (e.g. com.foo.login.user) into different shards. All the shards will be executed in parallel on the devices specified in the devices parameter.

REST API endpoint :

POST /app-automate/espresso/v2/build

Example cURL request :

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
-X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
-d '{"shards": {"numberOfShards": 2, "mapping": [{"name": "Shard 1", "strategy": "class", "values": ["com.foo.login.user", "com.foo.login.admin"]}, {"name": "Shard 2", "strategy": "class", "values": ["com.foo.logout.user"]}]}, "devices": ["Samsung Galaxy S20-10.0","Google Pixel 3-10.0"], "app": "bs://f7c874f21852ba57957a3fdc33f47514288c4ba4", "testSuite": "bs://e994db8333e32a5863938666c3c3491e778352ff"}' \
-H "Content-Type: application/json"

In the above example, the test cases will be split into 2 groups (i.e. shards). The first shard will consist of test cases belonging to com.foo.login.user and com.foo.login.admin classes. The second shard will consist of test cases belong to the com.foo.logout.user class. Both the shards will execute in parallel on Samsung Galaxy S20-10.0 and Google Pixel 3-10.0. This will block 4 parallel tests licenses in your App Automate subscription plan.

4. Use annotation strategy

Using this strategy, you can group test cases by annotations (e.g. com.foo.login.SmokeSuite) into different shards. All the shards will be executed in parallel on the devices specified in the devices parameter.

REST API endpoint :

POST /app-automate/espresso/v2/build

Example cURL request :

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
-X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
-d '{"shards": {"numberOfShards": 2, "mapping": [{"name": "Shard 1", "strategy": "annotation", "values": ["com.foo.login.SmokeSuite"]}, {"name": "Shard 2", "strategy": "annotation", "values": ["com.foo.login.RegressionSuite"]}]}, "devices": ["Samsung Galaxy S20-10.0","Google Pixel 3-10.0"], "app": "bs://f7c874f21852ba57957a3fdc33f47514288c4ba4", "testSuite": "bs://e994db8333e32a5863938666c3c3491e778352ff"}' \
-H "Content-Type: application/json"

In the above example, the test cases will be split into 2 groups (i.e. shards). The first shard will consist of test cases with com.foo.login.SmokeSuite annotation. The second shard will consist of test cases belong to the com.foo.login.RegressionSuite annotation. Both the shards will execute in parallel on Samsung Galaxy S20-10.0 and Google Pixel 3-10.0. This will block 4 parallel tests licenses in your App Automate subscription plan.

5. Use size strategy

Using this strategy, you can group test cases by specific test size i.e tests annotated with @SmallTest, @MediumTest or @LargeTest (e.g. small/medium/large) into different shards. All the shards will be executed in parallel on the devices specified in the devices parameter.

REST API endpoint :

POST /app-automate/espresso/v2/build

Example cURL request :

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
-X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
-d '{"shards": {"numberOfShards": 2, "mapping": [{"name": "Shard 1", "strategy": "size", "values": ["small", "medium"]}, {"name": "Shard 2", "strategy": "size", "values": ["large"]}]}, "devices": ["Samsung Galaxy S20-10.0","Google Pixel 3-10.0"], "app": "bs://f7c874f21852ba57957a3fdc33f47514288c4ba4", "testSuite": "bs://e994db8333e32a5863938666c3c3491e778352ff"}' \
-H "Content-Type: application/json"

In the above example, the test cases will be split into 2 groups (i.e. shards). The first shard will consist of test cases annotated with @SmallTest and @MediumTest. The second shard will consist of test cases annotated with @LargeTest. Both the shards will execute in parallel on Samsung Galaxy S20-10.0 and Google Pixel 3-10.0. This will block 4 parallel tests licenses in your App Automate subscription plan.

Device selection strategy

If you don’t want each of the shards to execute on all specified devices, you can set the deviceSelection attribute inside the shards parameter to any value. This will ensure that each shard will execute on any randomly selected device from the list of devices specified in the devices parameter. By default, its value is set to all.

REST API endpoint :

POST /app-automate/espresso/v2/build

Example cURL request :

curl -u "YOUR_USERNAME:YOUR_ACCESS_KEY" \
-X POST "https://api-cloud.browserstack.com/app-automate/espresso/v2/build" \
-d '{"shards": {"numberOfShards": 2, "deviceSelection": "any", "mapping": [{"name": "Shard 1", "strategy": "class", "values": ["com.foo.test1", "com.foo.test2"]}, {"name": "Shard 2", "strategy": "class", "values": ["com.foo", "com.foo.login"]}]}, "devices": ["Samsung Galaxy S8-7.0", "Google Pixel 3-10.0"], "app": "bs://f7c874f21852ba57957a3fdc33f47514288c4ba4", "testSuite": "bs://e994db8333e32a5863938666c3c3491e778352ff"}' \
-H "Content-Type: application/json"

In the above example, the test cases will be split into 2 groups (i.e. shards). Both the shards will execute in parallel, but each shard will execute on a single device randomly selected from the list (i.e. Samsung Galaxy S20-10.0 or Google Pixel 3-10.0). This will block 2 parallel tests licenses in your App Automate subscription plan (one for each shard).