Handle baseline variations for A/B testing

Review visual differences across multiple possible variations using the A/B Testing feature.

A/B testing allows you to create multiple snapshot variations and compare them to highlight differences across all variations. This helps handle intentional visual differences or non-deterministic elements without needing manual approval for every variation, streamlining your visual testing process.

Benefits

• Accelerate visual approvals: Automatically approve snapshots if they match any baseline variant.
• Simplify variation management: Manage all variants in one place without juggling multiple snapshot names.
• Increase testing accuracy: Confidently validate intentional visual differences in A/B testing scenarios.
• Reduce manual effort: Automate comparisons and approvals to save time for core product work.

Workflow

Step 1: Click the A/B testing icon to enable the A/B testing.

Step 2: Click the highlighted Create new thumps up icon to add a new baseline variant.

Step 3: Enter a name for the variant and click Create.

The baseline variant is added and available for comparison.

Variant management

Create and name variants

You can add new baseline variants from the A/B testing panel.

Rename variants

You can edit the name of an existing variant to improve clarity and organization. Click the edit icon next to the variant, enter a new name, and save your changes. Providing a name is required to complete the rename action.

Update variants

You can update an existing variant with the current snapshot.

• Click the highlighted arrow icon on a variant to replace it with the current snapshot and also approves it.
• When you approve a snapshot, it automatically replaces the matching variant.
• You can manually update a non-matching variant as well. This is useful when you don’t want to replace an existing matched variant.

Percy updates the default variant and shows the Default baseline variant updated notification.

Delete variants

Use the delete icon to remove added variants. This action does not affect past builds where the variants appear.

• You can delete variants only on the latest builds.
• You cannot delete a matching variant.

View variant metadata

See creation date, build number, default status, link to the build where it was created, and deletion status.

Things to keep in mind

• Each snapshot compares to all saved variants and the default baseline during build execution.
• A snapshot is automatically approved if it matches any of the available variants.
• If no exact match is found, Percy selects the variant with the smallest visual diff.
• If two variants have the same diff ratio with the head image, it matches with the default variant if available, otherwise, it follows the order in which the variants were created.
• You can assign or update which variant acts as the default for comparisons.
• Regions only apply when comparing a snapshot with the variant where that region was originally defined.
• During a merge, the system merges only the default variant and ignores the rest. You need to create the other variants again if required. Also, if a variant with the same name exists on the master for the same snapshot, it replaces that variant. If not, it replaces the default on master.
• You can switch to a non-matching variant in view-only mode to inspect its screenshot. However, approvals continue to apply only to the matching variant.
• You cannot approve a snapshot when multiple variants of the same snapshot are used as baselines across browsers. Update the baseline to a single variant to proceed with approval.

With support for baseline variations, Percy makes it easier than ever to manage A/B test scenarios and maintain reliable visual testing workflows—without added complexity.