How to perform Visual Regression Testing in NightwatchJS

Automation tools are primarily used to test web application functionality, but the modern web design trend has changed. The User Interface and User Experience (UI/UX) are getting the same importance as functionality and the best way to UI/UX elements is through Visual Regression Testing.

Visual Regression testing compares two screenshots, a baseline screenshot, and an actual screenshot. The baseline screenshot can be considered as an expected screenshot and the actual screenshot should always match the baseline screenshot. If it fails to match, the test can be marked as a fail.

NightwatchJS is a NodeJS-based open-source test automation framework that allows users to automate web applications with a bundle of features such as clear syntax, a built-in test runner, cloud testing support, page object support, and continuous integration. It supports the custom commands feature, where the user can define the custom commands and add the functionality for that command. Simply put, you can hook any plugins like node packages and use them in your automation scripts. Using this feature, we can automate the Visual Validation Testing in NightwatchJS.

This article will get into two different ways to perform Visual Regression Testing in NightwatchJS

• Nightwatch Visual Regression testing
• Nightwatch Visual Regression testing using Percy

Visual Regression Testing in NightwatchJS

The nightwatch-visual-testing is a third-party package that can be hooked into your NightwatchJS framework, and perform a Visual Regression Test.

Advantages of the plugin

• Tests can be configured and run on the local machine
• Comes with an ISC license so it is free to Use

Disadvantages of the plugin

• This plugin/package doesn’t provide any online services such as Review, Accept, Reject, etc.
• Configuring multiple browsers requires some setup and doesn’t come out of the box.
• Sharing report needs custom configuration or needs to be dependent on some other tool.
• The plugin is developed by individual contributors so there is no guarantee that it will be maintained for a long time.

Performing Visual Regression Testing using nightwatch-visual-testing package

Pre-Requisite: Set up Nightwatch JS

Step 1: Install the nightwatch-visual-testing package

The NightwatchJS doesn’t support Visual Testing out of the box, so we need to install the third-party package

Use the below command to install the package.

npm install nightwatch-visual-testing

Step 2: Configure NightwatchJS for Visual Testing

Open the nightwatch.conf.js file in your project, and add the configuration below.

• Add the custom_commands_path

Look for the variable custom_commands_path in nightwatch.conf.js add an additional value to that.

custom_commands_path: ['./node_modules/nightwatch-visual-testing/commands'],

• Add the custom_assertions_path

Look for the variable   custom_assertions_path in nightwatch.conf.js add an additional value to that.

custom_assertions_path: ['./node_modules/nightwatch-visual-testing/assertions'],

• Add the Screenshot configurations

Search for the     default: { inside the   test_settings: { block and add the screenshot folder locations

 globals: {
visual_regression_settings: {
outputDir: './tests_output',
threshold: 0.5
},
},

The threshold is the tolerance of pixels when comparing the two screenshots; the higher the value lower the sensitivity. 

Step 3: Create a Nightwatch Visual Test

Create a fresh javascript file inside your desired folder ( ex: tests/demoVRT.js) and write the simple code to test.

Let’s test the Browserstack home page i.e https://www.browserstack.com/

Example Code Snippet:

//demoVRT.js
describe('Nightwatch Visual Test Demo', function () {
it('Nightwatch Visual Test', async function (browser) {
await browser
.navigateTo('https://www.browserstack.com/')
.assert.compareScreenshot('First test', 'Screenshot captured!')
.end();
});
});

Step 4: Execute the Test

Execute Nightwatch Visual Validation test using the below command\

Syntax:

npx nightwatch <path_to_test_file>

Example:

 
npx nightwatch tests/demoVRT.js

Step 5: View the result

Once you execute the script the output will be shown in the command line terminal.

If there is no difference in the screenshots, then the test will be marked as Pass.

If there is any difference in the image, then the test fails

The folder tests_output/visual-regression contains multiple images, the actual image, the base image, and the difference between the actual and base image.

Visual Testing for Single HTML Element

The plugin also supports Visual Testing for a single HTML element with the command compareElementScreenshot(). 

Syntax: 

compareElementScreenshot('<selector>', '<name', 'message')

Example code Snippet:

//demoVRT.js
describe('Nightwatch Visual Test Demo', function () {
it('Nightwatch Visual Test', async function (browser) {
await browser
.navigateTo('https://www.browserstack.com/docs')
.assert.compareElementScreenshot('#product-menu-toggle','First test', 'Screenshot captured!')
.end();
});
});

Performing Nightwatch Visual Regression Testing with Percy

Percy is a popular visual testing tool that supports the visual testing of multiple browsers. It provides a dedicated build dashboard for each build, which helps the users view the test results online, and also provides functions such as Review, Accept and Reject the build. Percy supports both Manual and Automated visual testing. 

Advantages of Percy

• Percy’s visual testing reports can be viewed online.
• One can perform a review online and take action such as accept, reject or request changes.
• Percy has an inbuilt comparison mechanism for all browsers so rendered screenshots can be viewed for multiple browsers
• Percy supports both component and Page based visual testing.
• Percy allows you to adjust the diff sensitivity (same as screenshot threshold)

A step-by-step guide to Performing Visual Regression testing Nightwatch and Percy

Pre-Requisite: Set up Nightwatch JS

Step 1: Instal Required Percy packages

The Percy packages need to be installed to use the Percy commands in your project

Command to install Percy packages.

npm install --save-dev @percy/cli @percy/nightwatch

The above command installs the two packages @percy/cli and @percy/nightwatch. The @percy/nightwatch provides the custom commands for visual comparison and @percy/cli provides functionality to execute the test and upload to Percy infrastructure.

Step 2: Configure Percy for Nightwatch

• Import the Percy package

Open the nightwatch.conf.js file and add the required statement

const percy = require('@percy/nightwatch');

Note: The above require statement should be before module.exports

• Add the custom command

In your nightwatch.conf.js file search for custom_commands_path: and add the Percy command like below.

custom_commands_path: [percy.path],

Step 3: Create a simple Percy Nightwatch script

//demoVRT.js
describe('Nightwatch Visual Test Demo', function () {
it('Nightwatch Visual Test', async function (browser) {
await browser
.navigateTo('https://www.browserstack.com/docs')
.percySnapshot('Demo snapshot')
.end();
});
});

In the above code, we are navigating to the BrowserStack Documentation page and taking the screenshot for visual comparison.

The percySnapshot(‘Demo snapshot’) command  saves the screenshot with the name ‘Demo snapshot’

Step 4: Copy the PERCY_TOKEN

• Login to Percy (If you don’t have an account, create one by Signing Up)
• Create a New Project
• Navigate to Project Settings
• Copy the PERCY_TOKEN

Step 5: Set the PERCY_TOKEN environment variable

Based on the command-line tool you are using, you can set the environment variable as below.

# Unix
$ export PERCY_TOKEN="<your-project-token>"

# Windows
$ set PERCY_TOKEN="<your-project-token>"

# Powershell
$ $Env:PERCY_TOKEN="<your-project-token>"

Step 5: Execute Percy Test

Use the below command to execute your Percy NightwatchJS script.

npx  percy exec -- nightwatch <testscript_path>

Example:

npx  percy exec -- nightwatch tests/demoVRT.js

Step 6: View The Result

Once the execution is completed, the command line tool provides the output with Percy build URL. You can easily navigate to the build using the build URL to see the visual validation result.

If there are no changes, then Percy shows the message “No Changes”

If there is a difference between the baseline screenshot and the actual screenshot when Percy shows the side-by-side difference

HTML Element Visual Testing using Percy

Percy can also help you to capture and compare individual HTML elements. Instead of capturing the entire page you can capture the snapshot of the required element and perform visual testing. 

The scope option in percySnapshot() command can help to capture the HTML element

Example Code:

//demoVRT.js
describe('Nightwatch Visual Test Demo', function () {
it('Nightwatch Visual Test', async function (browser) {
await browser
.navigateTo('https://www.browserstack.com')
.percySnapshot('Single Element Snapshot', { scope: '#product-menu-toggle' })
.end();
});
});

In the above code, we are specifying the selector in the scope option instead of capturing the entire page. 

.percySnapshot(‘name’, { scope: ‘#product-menu-toggle’ }) : The HTML element which matches the selector  with id product-menu-toggle will be captured and used for Visual Validation. 

Percy NightwatchJS Best Practices

• Use the Visual Testing command for each web application page
• Schedule the build and analyze the result more frequently. This guarantees no accidental changes have been pushed.
• Use the diff sensitivity settings, such as strick, sensitive and relaxed mode, which helps to identify the color differences.
• Consider testing responsive visual testing, The responsive visual testing ensures there are no visual bugs in different screen resolutions. This can be done in by choosing the available resolution in Percy build dashboard
• The Multiple browser testing helps to ensure the page is rendering in all the required browsers without any glitches. Percy allows users to choose multiple browsers in Project settings.

Conclusion

Manual testing is always error-prone whereas the automated way of visual testing saves time and provides the best result. The major problem when you use a different tool is upgrading the skill set, but Percy provides out-of-the-box integration for all major automation frameworks, so the learning curve is very less. Also, using Percy is very easy.

By integrating Percy with NightwatchJS, the single test automation framework can be enhanced to perform both functional and Visual regression testing.

Get Started Free