Python Visual Regression Testing: Tutorial

Visual testing is an essential aspect of software testing that focuses on verifying the graphical user interface (GUI) of an application. It involves checking whether the GUI components across different viewports.

Visual testing is usually performed manually by human testers who visually inspect the application and compare its actual appearance with its expected appearance. 

Why is Automated Visual Testing necessary?

According to a report, the global market for automated software testing is expected to grow to $52.7 billion by 2027, at a CAGR of 16.4% during the forecast period. 

With modern software applications’ increasing complexity and diversity, manual visual testing can be time-consuming, error-prone, and inefficient.

To address these challenges, automated visual testing has emerged as a promising solution that leverages computer vision and machine learning techniques to perform visual testing automatically and reliably. 

Automated visual testing tools use image comparison algorithms to capture screenshots of the application at different stages of its development and compare them with the expected screenshots to detect any visual regressions or inconsistencies.

Automated visual testing offers several benefits over manual visual testing, including

• Increased speed and efficiency: Perform hundreds or even thousands of visual tests in a fraction of the time it takes for a human tester to do the same.
• Improved accuracy and coverage: Detect visual bugs that human testers may miss due to fatigue, subjectivity, or human error.
• Better scalability and repeatability: Integrate into a continuous integration/continuous delivery (CI/CD) pipeline and can run tests repeatedly and consistently across different platforms and environments.
• Cost-effectiveness: Reduce the cost of testing by minimizing the need for manual testing and increasing the efficiency of the testing process.

What is Python and how does it relate to Automated Visual Testing?

Python is a popular high-level programming language introduced in the late 1980s. It is known for its simplicity, versatility, and readability, making it a popular choice for software development across various domains such as web development, data science, artificial intelligence, and automation.

• One of the reasons why Python is so popular is its ease of use and learnability. 
• Its syntax is concise and intuitive, making it easy for developers to write clean and maintainable code. 
• Python has a large and supportive community of developers who have created a vast library of modules and packages that can be easily integrated into Python applications to add additional functionality.
• It is widely used in software development is its compatibility with many other programming languages and platforms.
• It can be easily integrated with other programming languages such as C++, Java, and .NET, making it a powerful tool for developing complex apps.
• Additionally, Python is cross-platform and can run on OS such as Windows, Linux, and macOS.
• Developers can also use Python as a testing environment for automated visual testing. 
• There are several Python-based testing frameworks such as Robot Framework, and that offer built-in support for automated visual testing. 
• These frameworks provide a set of libraries and APIs that enable developers to write visual test cases and assertions using Python code, and run them automatically as part of their test suite.

Setting up Python Automated Visual Testing

Set up required for performing visual regression testing Python will largely depend upon what is the structure of application and nature of tests to be performed. So depending on the type of platform it could be a web testing set up or a mobile app testing set up. 

Let’s take the example of web testing, web testing can be looked at from a broad perspective using a input automation tool or from a close perspective using a browser automation tool.

Tools needed for Automated Visual Testing

Several products and libraries are available for developers to build visual testing workflows in Python. Here are some popular options:

5. PyTest: Pytest provides a robust testing framework with a flexible test runner and a wide range of testing features. Using the Pytest framework for visual testing in Python, you can easily test your application’s graphical user interface, ensuring that it is visually consistent and functional.
6. Behave: It is a behavior-driven development (BDD) framework. Behave uses natural language to describe the behavior of an application in a test scenario, making it easy to write and understand tests. Behave also offers support for automated visual testing using Python and tools like Selenium and Appium.
7. PyAutoGUI: PyAutoGUI is a Python library that allows developers to automate tasks by controlling the mouse and keyboard. It can be used for GUI automation and visual testing. PyAutoGUI can capture screenshots and compare them with expected images, making it suitable for building visual testing workflows.
8. Selenium WebDriver: Selenium WebDriver is a popular open-source testing framework for web applications. It provides a Python API that allows developers to automate browser actions, capture screenshots, and perform visual comparisons. Selenium WebDriver can be used to write visual test cases for web applications that require cross-browser and cross-platform testing.
9. Robot Framework: Robot Framework is a generic open-source testing framework that supports various test automation needs. It provides a Python API for writing test cases and keywords, including visual testing keywords. Robot Framework can be used to create end-to-end testing workflows that incorporate visual testing with other types of testing.

And then on top of this, they are general and not specific enough for a testing person to start building test cases directly; they would have to do a lot of boilerplates set up with these kinds of tools before they could start integrating their testing process in a production environment.

Hence, for automated visual regression testing of a website with Python, we will use BrowserStack Percy, which provides advanced functionality like snapshots, automatic diff and scheduling, making the testing workflow much more powerful.

Start Testing with Percy

Installation and configuration of tools

To get started, you’ll need to create a local web server to deploy your site on. Once your server is up and running, you can install the Percy CLI to take automated visual snapshots of your site.

To install the Percy CLI, open your terminal and run the following command:

npm install --save-dev @percy/cli

Next, you’ll need to install the Percy Python package, percy-selenium. You can do this by running the following command in your terminal:

pip install percy-selenium

Setting up the Test Environment

Once you have the CLI and the Python package installed, you can start setting up your test environment. Create a new Python script and import the percy_snapshot function from the Percy module:

from percy import percy_snapshot

Next, you’ll need to create a function that takes snapshots of your website. You can do this by using the percy_snapshot function and passing in the webdriver instance and a snapshot name.

browser = webdriver.Firefox()
browser.get('http://example.com')
​
# take a snapshot
percy_snapshot(browser, 'Python example')

Finally, you’ll need to set up your Percy access token. To do this, run the following command in your terminal:

$ export PERCY_TOKEN=[your-project-token]

This will set your access token as an environment variable, allowing you to use Percy to take automated visual snapshots of your site.

Creating Tests for Automated Visual Testing with Python

Now that you have set up Percy CLI for visual testing on your local machine, it’s time to start creating test cases and running them automatically using Python.

Writing Test Cases for Automated Visual Testing

To write test cases for automated visual testing, you need first to identify the areas of your website that need to be tested. 

• You can test individual elements or entire pages and test for things like layout, font, and color consistency. 
• Once you have identified what you want to test, you can start writing test cases.
• To write a test case, you can use the percy_snapshot function that we used in the previous section. 
• This function takes a WebDriver instance and a snapshot name as arguments. 

For example:

percy_snapshot(browser, 'homepage')

This will take a snapshot of the current state of the browser and save it with the name “homepage”. You can use this function to take snapshots of your website at different states and compare them over time.

Let’s say for example, we want to carry out the automation of the following test activity:

• Open bstackdemo.com
• Select the first item and add it to the cart
• Click on checkout
• Take a snapshot

Here’s how you can create a test case for this scenario using Percy Python:

from selenium import webdriver
from percy import percy_snapshot


def test_bstackdemo_checkout():
browser = webdriver.Chrome()
browser.get("https://bstackdemo.com")
browser.find_element_by_class_name("text-center").click() # select first item
browser.find_element_by_xpath("//button[text()='Add to Cart']").click()
browser.find_element_by_id("cart").click() # click on checkout
percy_snapshot(browser, "Checkout") # take a snapshot
browser.close()

Understanding Python Libraries for Automated Visual Testing

Percy Python is a Python library that provides a wrapper around the Percy API. This library allows you to interact with the Percy API in Python code, making it easy to automate visual testing.

To use Percy Python effectively, you need to understand:

Where to take snapshots?

You should take snapshots of pages that are critical to the user experience or have frequent changes. You can also take snapshots of pages with a high visual impact, such as landing pages or product pages.

Snapshot Configuration and Optimization

You can configure snapshots by specifying various parameters, such as the widths and heights of the viewport or the wait time before taking a snapshot. Optimizing snapshots can reduce the size of the snapshot, which can make the testing process faster and more efficient.

Diff Checking

Percy Python uses a diffing algorithm to detect visual regressions. It compares the new snapshot with the previous one and highlights any differences. You can configure the diffing threshold to control the diffing algorithm’s sensitivity.

Approvals and Reporting

After a snapshot is taken, you can approve or reject it, and add comments to it. You can also view reports showing your tests’ status, including the number of snapshots, the number of changes, and the overall test pass rate.

Utilizing Python libraries for Automated Visual Testing

To create effective tests, you should take snapshots of critical components on your website, such as the homepage, product pages, or checkout pages. 

You can also automate the snapshot process by integrating your tests with your code deployment process, such as with CI/CD tools like Jenkins or Travis CI.

Running Automated Visual Tests with Python

To ensure your tests are running correctly, you should maintain and run the test scripts regularly. You can also integrate the test scripts with your code deployment process to run automated visual tests with code deployment.

Run the script using the following commands :

$ export PERCY_TOKEN=[your-project-token]
$ percy exec -- [python test command]
[percy] Percy has started!
[percy] Created build #1: https://percy.io/[your-project]
[percy] Snapshot taken "Python example"
[percy] Stopping percy...
[percy] Finalized build #1: https://percy.io/[your-project]
[percy] Done!

Use the advanced configurable snapshot features to build in-depth testing processes.

Log on to the Percy Dashboard to see the results and work on build approval.

Understanding the Test Results

After running the tests, you can view the test results and collaborate with your team on the Percy dashboard. You can check the usage of the tests and monitor the builds to ensure that the visual consistency of your website is maintained.

Debugging and Resolving issues with Automated Visual Tests

If you encounter any issues with your tests, you can debug them by checking the logs and analyzing the code. You can also learn better test syntax and ensure test quality in the code to prevent issues from occurring in the first place.

7 Best Practices for Automated Visual Testing

Implementing efficient visual testing in Python requires a systematic approach and adherence to best practices. Here are seven best practices for implementing efficient visual testing in Python from a coder’s perspective:

1. Use Appropriate Image Comparison Algorithms

The choice of image comparison algorithm can greatly affect the accuracy and efficiency of visual testing. There are several image comparison algorithms available such as pixel comparison, structural similarity index (SSIM), and mean squared error (MSE). 

Developers should choose an appropriate algorithm based on the specific requirements of their testing environment.

2. Use Image Diffing to minimize False Positives

Image diffing is a technique to detect differences between two images. It can minimize false positives in visual testing by highlighting only the areas that have changed, instead of flagging the entire image as failed. 

3. Optimize Image Size and Format

Optimizing image size and format can improve the efficiency of visual testing. Large images can take a long time to process and compare, slowing down the testing process. Developers should ensure that images are of an appropriate size and format for their testing environment.

4. Organize and name Test Images appropriately

Organizing and naming test images appropriately can improve the maintainability and readability of test code. Developers should use descriptive and meaningful names for test images and organize them in a logical directory structure.

Talk to an Expert

5. Use Version Control for Test Images

Version control can help manage changes to test images and ensure that the correct version is being used in testing. Developers should use a version control system such as Git to track changes to test images and ensure that the correct version is being used in testing.

6. Use a Reliable Test Environment

The reliability of the test environment can affect the accuracy of visual testing. Developers should ensure that the test environment is stable and consistent, with minimal variations between test runs. Using virtualization technologies such as Docker can help ensure the consistency of the test environment.

7. Integrate Visual Testing with other types of testing: Visual testing should be integrated with functional testing and performance testing. This can help ensure that the application functions correctly and performs as expected.

In conclusion, Python is a powerful and versatile programming language widely used in software development. It offers a range of features that make it an ideal choice for developing complex applications and testing environments. 

Developers who want to explore automated visual testing should consider using Python-based testing frameworks to write and execute their visual test cases.