Parallel test suites

Running complex test suites with Percy

Parallelism is a common way to speed up test suites by running multiple tests concurrently and then combining results at the end.

Percy has built-in support for complex test suites that run in parallelized CI services or parallel test runners. There are two different ways that you can setup Percy for parallel test suites: across many machines or running parallel tests on the same machine.

Percy is designed to support any parallelized test environment. Once set up, snapshots are pushed from your tests to Percy and grouped in the same Percy build, no matter if your tests are run in different processes or even on different machines.

Parallel suites split across different machines

If your tests are split across multiple machines or containers, there are two key steps to make it work with Percy:

• Pass a --parallel flag to the Percy CLI (percy exec --parallel -- [test command]).
• After all tests have completed, call percy build:finalize to finalize the entire Percy build.

Example config

This is not a working example, but it illustrates the steps to take.

How it works

Percy’s parallel support works by looking for two different environment variables to be set. If your CI provider is supported by Percy, PERCY_PARALLEL_NONCE will automatically be set for you.

• PERCY_PARALLEL_TOTAL - The number of parallel nodes being ran
• PERCY_PARALLEL_NONCE - A unique ID for this parallel build. Percy uses this to group many builds into a single build (with matching nonce’s) . It can only be used once.

Depending on the value of PERCY_PARALLEL_TOTAL, there are two different ways the Percy API behaves.

If PERCY_PARALLEL_TOTAL is set to a specific number, the API will wait for that many finalized builds to permanently finalize the Percy build. That means builds can no longer be created or added with that nonce once the number is reached. With this approach, you will not call percy build:finalize

If PERCY_PARALLEL_TOTAL is set to to -1 this tells the API to disregard the number of builds created and finalized. The API will then wait for a finalize all builds call, which will close the build for that nonce. The Percy CLI sets the total to -1 when the --parallel flag is passed.

Parallel tests on the same machine

If your tests are running in parallel on the same machine or container, the path to setting up Percy for parallel builds can be easier (as there’s no environment variables to setup).

Percy’s SDKs works by starting a locally running server (by default on http://localhost:5883). The SDKs then POST their captured snapshots to this server, which will then process the snapshots and send them to the Percy API. As long as this server is accessible while your tests run, Percy will capture snapshots. There are two different ways this approach can work, which depends on how your tests are setup.

If your test command manages spawning the parallel process, you can keep using percy exec as you were before. For example, a project using Ember Exam won’t need to setup any extra configuration since Ember exam opens many browsers on the same machine to split the tests up.

If you are managing the processes on your own, you would use the Percy start CLI command (percy exec:start rather than percy exec) to start a local Percy server. This will create a Percy build and wait for snapshots to be sent to the Percy server (which happens when percySnapshot is called). Percy only requires a single server to be open and accepting snapshots. Once all of your tests have completed, you will need to call percy exec:stop to finalize the build (which closes the running Percy server). This should be run only after all the tests have completed & exited to ensure all snapshots have been sent.

Example config

This is not a working example, but it illustrates the steps to take.

Troubleshooting

Setting PERCY_PARALLEL_NONCE

If your CI provider is not supported by Percy out of the box, there’s one environment variable you will need to set for parallel builds to work: PERCY_PARALLEL_NONCE. This environment variable should be set to an ID that is only used once, ever. Typically using a workflow ID or CI build number works well.

If your CI provider reuses these values on rebuilds, it will cause issues with Percy (and you should find a more unique nonce to use, instead).

Stuck in receiving

If a build is hanging in the “receiving” state, even after tests have finished, that means the build was never fully finalized.

If PERCY_PARALLEL_TOTAL is set to a specific number, make sure there were that many Percy builds finalized in the test run. For example, if PERCY_PARALLEL_TOTAL=4 and there were only 3 builds ran/finalized, Percy will hang in a receiving state. This is because Percy is waiting for 4 finalized builds.

If PERCY_PARALLEL_TOTAL is set to-1 (using --parallel), make sure percy build:finalize was called. If it was, make sure PERCY_PARALLEL_NONCE is the same for the tests and the finalize step.

Advanced NONCE setting

If Percy has support for your CI provider, we will set a default NONCE for you. This makes it so parallel builds can work without any extra setup or needing to understand the finer details of parallel builds. Most of the time this will work without any issues, but some CI providers will reuse the same value we are setting the NONCE to on rebuild. For example, GitHub actions reuse the workflows GITHUB_RUN_ID. This will cause the Percy build to fail with an error saying you can’t add new snapshots to an already finalized build (this is because the NONCE matches an existing build and all NONCEs need to be unique).

To work around this problem, you will need to create your own NONCE or find a NONCE that your CI provider gives which is unique for each build.

GitHub Actions

For GitHub actions, you will need to create a NONCE that’s unique for each test run and then persist it across the workflows. This can be done by using GitHub Actions set-output parameters to pass the newly created NONCE between jobs in the workflow run.

This example uses the existing GITHUB_RUN_ID variable we use by default, but it appends a date to ensure it’s unique for each test run. See output parameters & jobs.<job_id>.outputs for more information.

Other CI

For any other CI, you can always open a GitHub discussion on the Percy CLI repo.