How to Perform Visual Regression Testing using WebdriverIO

Visual testing is a type of testing to verify that the visual aspects of the software are not breaking, basically, recording the UI bugs. In general, regression testing is to verify that the new code changes are not affecting the existing features. In this guide, we’ll showcase a step-by-step tutorial on performing visual regression testing using WebDriverIO. 

What is Visual Regression Testing?

Visual Regression, to be specific, is intended to check that any code changes did not affect the UI by comparing screenshots from before and after. Visual regression tests are significant in identifying whether the visual errors have occurred after making changes to the code. It can be integrated into the existing test framework like WebdriverIO using tools that support visual tests.

Visual Regression Testing with WebdriverIO Framework

WebdriverIO is an effective tool to perform visual testing as it provides an API for image comparison. Wdio-image-comparison-service is one of the services provided by WebdriverIO to compare images in various platforms such as browsers, mobile browsers, and hybrid applications.

Run Visual Regression Test using WebdriverIO

To begin writing the test cases, we need to install a few prerequisites.

1. Node.js
2. WebdriverIO

Additionally, we should install the wdio-image-comparison-service . Pass the following command in the root directory of the project to install the image comparison service.

npm install -save -dev wdio-image-comparison-service

To ensure whether the package has been installed properly, we can verify by checking the service has been added in the package.json file

Configuration 

Wdio-image-comparison-service is a normal service that can be used as any other service. To utilize the service, we will need to add a few snippets to the wdio.config.js file

const { join } = require('path');
// wdio.conf.js
exports.config = {
// ...
// =====
// Setup
// =====
services: [
['image-comparison',
// The options
{
// Some options, see the docs for more
baselineFolder: join(process.cwd(), './tests/baselineBuild'),
formatImageName: '{tag}-{logName}-{width}x{height}',
screenshotPath: join(process.cwd(), '.tmp/'),
savePerInstance: true,
autoSaveBaseline: true,
blockOutStatusBar: true,
blockOutToolBar: true,
// NOTE: When you are testing a hybrid app please use this setting
isHybridApp: true,
// Options for the tabbing image
tabbableOptions:{
circle:{
size: 18,
fontSize: 18,
// ...
},
line:{
color: '#ff221a', // hex-code or for example words like `red|black|green`
width: 3,
},
}
// ... more options
}],
],
// ...
};

Writing Tests

Now we have our basic setup done for our image comparison, let us write a test case to understand how the wdio service helps in comparing images.

We are using the Mocha framework to write our test cases. We’ll wrap them in a description and it blocks. Since WebdriverIO supports only asynchronous operations, we will also write our cases in async.

describe("Example", () => {
beforeEach(async () => {
await browser.url("https://www.google.com/");
});

it("should save some screenshots", async () => {
// Save a screen
await browser.saveScreen("examplePaged");

// Save an element
await browser.saveElement(await $("//input[@name='q']"));

// // Save a full page screenshot
await browser.saveFullPageScreen("fullPage");

// // Save a full page screenshot with all tab executions
await browser.saveTabbablePage("save-tabbable");
});

it("should compare success with a baseline", async () => {
// Check a screen
await expect(await browser.checkScreen("examplePaged")).toEqual(0);

// Check an element
await expect(await browser.checkElement(await $("//input[@name=`q`]"))).toEqual(0);

// Check a full page screenshot
await expect(await browser.checkFullPageScreen("fullPage")).toEqual(0);

// Check a full page screenshot with all tab executions
await expect( await browser.checkTabbablePage("check-tabbable")).toEqual(0);
});
});

In the above example, we have added the navigate to url command in the before block. In our test, we have two “it” blocks. The first “it” block saves the screenshots to the commands that we are passing and the second “it”  block compares the actual and expected images.

Functions available in WebdriverIO for Image Comparison

1. browser.saveScreen(“ ”);

This command will save the whole screen that is in the view. It will neither scroll down or up, just capture the current screen. 

Example:

await browser.saveScreen("examplePaged");

await expect(await browser.checkScreen("examplePaged"})).toEqual(0);

The first line saves the screen and the checkScreen compares the saved screen in the baseline to the current screen that the framework is capturing.

2. browser.saveElement(“elementId”);

This command will capture and save a particular element that we wish to capture. We pass the locator in the function.

Example:

await browser.saveElement(await $("//input[@name='q']"));

await expect(await browser.checkElement(await $("//input[@name=`q`]")).toEqual(0);

In the above example, the first line saves a screenshot of an element with the given locator. Whereas, the second line expects the saved element and checks using the checkElement command.

3. browser.SaveFullPageScreen();

This command will capture and save the full page. It will scroll down and up to capture everything that is present in the present screen and save them.

Example:

await browser.saveFullPageScreen("fullPage");

await expect(await browser.checkFullPageScreen("fullPage")).toEqual(0);

From the above example, the first line accepts an argument to save the full page screen. The second line checks whether the saved screen and the current screen are equal using the checkFullPageScreen command.

4. browser.saveTabbablePage();

The above command saves a full-page screenshot inclusive of all the tab executions.

Example:

await browser.saveTabbablePage("save-tabbable"});

await expect(await browser.checkTabbablePage("check-tabbable")).toEqual(0);

From the above example, the first line saves the tab executions. The second line captures and compares the tabs in the current page using the checkTabbablePage command.

Baseline images are saved in the folder named baselineBuild. The test executed images are captured in .tmp folder which contains both actual and diff folders which compare the actual and different images.

Using Percy tool for Visual Testing

Percy by BrowserStack is visual testing as a service platform. Percy.io enables you to review visual changes in your application and offers a toolkit for you to get started with visual testing in whatever framework you are using.

Setup

1. Ensure you have Node.js installed in your computer
2. Ensure to have WebdriverIO installed. 
3. Run the following command from your root work directory to install Percy.

npm install --save-dev @percy/cli @percy/webdriverio

After installing all the packages, your package.json should be something like this.

Try Percy for Free

Performing Visual Regression Testing using WebdriverIO and Percy

To begin with, we need to create a project in Percy. After logging in with Percy, click on the Create Project and enter the project name.

Once you create the project, you will land on the project dashboard. You will see a unique project token generated. Pass the token in your terminal to integrate the project with Percy.

Writing Tests with Percy

Now, let’s write our test case to compare the screenshots with Percy.

const percySnapshot = require("@percy/webdriverio");

describe('webdriver.io page', () => {
it('should have the right title', async () => {
await browser.url('https://webdriver.io');
await expect(browser).toHaveTitle('WebdriverIO · Next-gen browser and mobile automation test framework for Node.js | WebdriverIO');
await percySnapshot(browser, 'webdriver.io page');
});
});

Initially, we need to require Percy in our test file. We are writing our test case using Mocha with describe and it block. We are navigating to the WebdriverIO URL and expecting the title to have WebdriverIO. With the percySnapshot command, we are capturing the screenshot. percySnapshot accepts three arguments.

 percySnapshot(browser, <snapshotName>, <options>)

Running the Test

Now, let us run our Percy test case. For it, we need to configure the command in our package.json file.  Add this command under scripts.

"test:percy": "percy exec -- wdio run wdio.conf.js"

To run the test, pass npm run test:percy command in your terminal. Once you run the test, you should see the Percy running command and generate a link to the Percy.

In the above screen, you can see a link (Finalized build #9) has been generated. When you open the link in the browser, you can see the snapshot of the test in the Percy dashboard. It should look something like this.

Differences in running with WebDriverIO only vs WebDriverIO + Percy

Now, we saw how to perform Visual testing using WebdriverIO and Percy. We’ll compare both the services to understand how they are useful in Visual testing. 

WebdriverIO for Visual Regression Testing

Pros

• Service is maintained actively
• Open-source
• Feature-rich (Allows customized testing types)
• Backed by OpenJS Foundation

Cons

• Requires external boilerplate for StoryBook
• The reporting for the WebdriverIO image comparison service is not very clear and it might be difficult to identify the actual and different results.

WebDriverIO + Percy for Visual Regression Testing

Pros

• Actively maintained and updated
• Easy Storybook compatibility
• Owned by BrowserStack
• Compatible with several testing frameworks like Selenium, Cypress, WebdriverIO, NightWatch, NightmareJS, Puppeteer, and many more.
• Clear and excellent reporting ( easy to compare the actual and different results)
• Cross-browser visual testing

Cons

• It is quite expensive when we opt for business-level enterprise

There you have it. A detailed tutorial that will help you perform visual regression testing either with WebDriverIO along or with WebdriverIO + Percy. You can definitely go for the one which suits your overall testing needs but with Percy, you can integrate automated visual reviews into extensive processes, frameworks, and libraries. The extra level of automation will give your QA team total control over preventable mistakes while saving time and resources.

Try Visual Regression Testing on Percy