App & Browser Testing Made Easy

Give your users a seamless experience by testing on 3000+ real devices and browsers. Don't compromise with emulators and simulators

Home Guide Appium Visual Testing: The Essential Guide

Appium Visual Testing: The Essential Guide

By Pawan Kumar, Community Contributor -

Table of Contents

Appium is a free, open-source, cross-platform testing tool for desktop, mobile, and desktop apps. 

  • It was first developed to automate iOS and Android mobile apps and has developed into a fully functional platform offering WebDriver-based automation options. 
  • It supports app automation across platforms like iOS, Android, and Windows. 
  • Since version 2.0, every driver has been isolated from the Appium server apps and can be managed alone using the Appium Driver command line interface.

What is Visual Testing?

Visual testing is a crucial type of testing that may identify both visual defects and functional defects. Although it is generally complementary to functional testing, visual testing is combined.

  • Visual testing assesses visual changes to ensure that only deliberate UI changes have been implemented in production. 
  • Many businesses mistakenly believe that visual testing is not required while conducting thorough functional tests. 
  • Visual testing finds flaws that functional testing misses and verifies that your program looks as it should
  • Visual regressions are equally important as functional regressions, especially when they reveal issues that a functional test suite might miss.

Benefits of Visual Testing

No matter how well your application’s functionality works, it will be useless to the consumer unless it can give a decent user interface and user experience. Nowadays, we talk about UI (User Interface) and put a lot of emphasis on UX (User Experience).

  • When we give a better user experience, the visual testing suite becomes much more crucial since a more appealing visual design may be recognized as more usable. 
  • The benefits of automating visual tests are that they provide long-term cost efficiency, are faster than manual tests, are more precise because they cannot include human issues, and deliver pixel-perfect visual tests.
  • They are reusable and clear because they offer automatic reports that are easily and readily accessible by anyone on the team.

Given its importance, businesses should devote substantial time and resources to visual testing. Combined with current developer tests, it will aid in detecting visual problems in the early phases of the development lifecycle.

Why perform Appium Visual Testing?

We know how to take screenshots using Appium. But how do we balance two screenshots to detect visual differences?

A basic approach to the visual distinction would be far too greedy because even condensation algorithms may produce nearly undetectable differences. In the realm of open-source software, OpenCV is an assortment of image manipulation capabilities that can be used without the user needing to comprehend the intricate details of its inner workings. Appium has OpenCV support, but it is not enabled by default since developing OpenCV and its Node.js bindings takes time and must be done on each platform independently. 

For the easiest way to get everything ready for usage with Appium, run npm install -g opencv4nodejs.

Steps for Visual Validation

The primary goal is to snap screenshots of each view we encounter while traveling through our app (maybe during a functional test). We compare these screenshots to a prior screenshot of the identical view using an image analysis application. 

We may have detected a visual regression if there are any significant changes. At that point, we may either reject the test or record the difference in a database for further assessment by the team.

Step 1: In this step, We’ll need to test the application and take screenshots.

Step 2: In this step, These screenshots are compared to the baseline screenshots by the automation tool. Typically, baseline screenshots are photographs a tester has confirmed were taken during previous test sessions.

Step 3: After getting the results of the image comparisons, the application generates a report detailing all the discrepancies observed.

Step 4: In the last step, The tester reviews the report, determining if each difference is a bug or a valid change (false positives). The baseline photographs are updated as a result of this.

You don’t have any baseline photos for the first test run. As a result, photographs from the first run are frequently utilized as baseline images. They are compared to the screenshots starting with the following run.

Visual Comparison Example

We are validating the “$” sign in this image because $ is not having your locator. First, we take a screenshot of the UI and then use OpenCV, to compare the images if the images are not the same then we will raise a bug to the responsible team.

Visual Comparison Example

Setting up the Appium Visual Testing Environment

Setup Appium

Download and install Java (JDK) and set a JDK and bin folder path.

  • Download the “.exe” file from here (Version: jdk1.8.0_91 or whichever is the latest you find there).
  • Install the “.exe” file.
  • Setup the JDK bin folder path in your system’s environment variable.

 Download the Android SDK

  • Click on the link “android-sdk_r24.4.1-windows.zip” (or whichever is the latest you find there) and click the download button.
  •  Once the zip file gets downloaded, unzip the folder.
  • Now click on the “SDK Manager.exe” file
  • This opens the Android SDK Manager window. Select “Tools” and the Android platform on which you will perform your tests.

Install Appium

  • Open the Appium Official link.
  • Click on the Download link.
  • Select the OS of the machine you are working on and download the appropriate version.
  • Unzip the downloaded zip folder.
  • Install the .exe file “appium-installer”.

Setting up Appium for visual testing

Appium has support for OpenCV, albeit not enabled by default since developing OpenCV and its Node.js bindings takes time and must be done on specific systems.

  • The simplest method to get things ready for usage with Appium is to run npm install -g opencv4nodejs
  • This will try to install the Node bindings globally and download and build OpenCV on your machine. 
  • If that doesn’t work, you may install OpenCV via Homebrew and then install the Node bindings with the OPENCV4NODEJS_DISABLE_AUTOBUILD=1 env flag in front to instruct it to use the system-installed binaries.

Once you’ve installed the opencv4nodejs package, you must ensure it is accessible to Appium when it starts. One method is to execute the npm install command without the -g parameter within the Appium directory. Another option is to include your global node_modules folder in the NODE_PATH environment setting.

Performing Appium Visual Testing

Test Case

Follow the below test steps for visual testing:

  1. First we store an image for the app home page as a baseline image.
  2. Launch the app using appium cap file. 
  3. Wait to upload the homepage
  4. Click to Add an item to the cart 
  5. Click on the cart button 
  6. Go to the cart and verify the cart screen with the baseline image (ScreenShot)

Homepage add to cart

In the first execution, it will same as the baseline images at the given path. And in the second execution its returns a match threshold value which we already set up in our code.

Framework Design and Code

For framework design, follow below structure

FrameworkDesign

Test Script 

AppiumVisualTestBrowserStackAPP.java

import io.appium.java_client.MobileBy;
import io.appium.java_client.imagecomparison.SimilarityMatchingOptions;
import io.appium.java_client.imagecomparison.SimilarityMatchingResult;
import java.io.File;
import java.net.URISyntaxException;
import org.apache.commons.io.FileUtils;
import org.junit.Test;
import org.openqa.selenium.By;
import org.openqa.selenium.OutputType;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;


public class AppiumVisualTestBrowserStackAPP extends BaseTest {

// need somewhere to store match files; change this path according to your location
private final static String Validation_path = "/Users/username/Desktop/";

private final static String CHECK_HOME = "home_screen";
private final static String CART_PAGE = "cart_page";
private final static String BASELINE = "BASELINE_";

private final static double MATCH_THRESHOLD = 0.99;

private final static By ADD_TO_CART = MobileBy.AccessibilityId("add-to-cart-12");
private final static By NAV_TO_CART = MobileBy.AccessibilityId("nav-cart");

@Override
protected DesiredCapabilities getCaps() throws URISyntaxException {
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("platformName", "Android");
capabilities.setCapability("deviceName", "Android Emulator");
capabilities.setCapability("automationName", "UiAutomator2");
capabilities.setCapability("app", getResource("apps/browserstack-demoapp.apk").toString());
// make sure we uninstall the app before each test regardless of version
capabilities.setCapability("uninstallOtherPackages", "io.cloudgrey.the_app");

return capabilities;
}

private WebElement waitForElement(WebDriverWait wait, By selector) {
WebElement el = wait.until(ExpectedConditions.presenceOfElementLocated(selector));
try { Thread.sleep(750); } catch (InterruptedException ign) {}
return el;
}

@Test
public void testAppDesign() throws Exception {
WebDriverWait wait = new WebDriverWait(driver, 5);

// wait for an element that's on the home screen
WebElement addToCart = waitForElement(wait, ADD_TO_CART);

// now we know the home screen is loaded, so do a visual check
doVisualCheck(CHECK_HOME);

// Click on add to cart btn for adding item in card
addToCart.click();
WebElement navToCart = waitForElement(wait, NAV_TO_CART);

//click to cart btn
navToCart.click();

// perform our second visual check, this time of cart page
doVisualCheck(CART_PAGE);
}

private void doVisualCheck(String checkName) throws Exception {
String baselineFilename = Validation_path + "/" + BASELINE + checkName + ".png";
File baselineImg = new File(baselineFilename);

// If no baseline image exists for this check, we should create a baseline image
if (!baselineImg.exists()) {
System.out.println(String.format("No baseline found for '%s' check; capturing baseline instead of checking", checkName));
File newBaseline = driver.getScreenshotAs(OutputType.FILE);
FileUtils.copyFile(newBaseline, new File(baselineFilename));
return;
}

// Otherwise, if we found a baseline, get the image similarity from Appium. In getting the similarity,
// we also turn on visualization so we can see what went wrong if something did.
SimilarityMatchingOptions opts = new SimilarityMatchingOptions();
opts.withEnabledVisualization();
SimilarityMatchingResult res = driver.getImagesSimilarity(baselineImg, driver.getScreenshotAs(OutputType.FILE), opts);

// If the similarity is not high enough, consider the check to have failed
if (res.getScore() < MATCH_THRESHOLD) {
File failViz = new File(Validation_path + "/FAIL_" + checkName + ".png");
res.storeVisualization(failViz);
throw new Exception(
String.format("Visual check of '%s' failed; similarity match was only %f, and below the threshold of %f. Visualization written to %s.",
checkName, res.getScore(), MATCH_THRESHOLD, failViz.getAbsolutePath()));
}

// Otherwise, it passed!
System.out.println(String.format("Visual check of '%s' passed; similarity match was %f",
checkName, res.getScore()));
}

}

BaseTest.java 

import io.appium.java_client.android.AndroidDriver;
import java.io.IOException;
import java.net.URISyntaxException;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Base64;
import org.junit.After;
import org.junit.Before;
import org.openqa.selenium.remote.DesiredCapabilities;

public class BaseTest {

AndroidDriver driver;

protected DesiredCapabilities getCaps() throws Exception {
throw new Exception("Must implement getCaps");
}

@Before
public void setUp() throws Exception {
URL server = new URL("http://localhost:4723/wd/hub");
driver = new AndroidDriver(server, getCaps());
}

@After
public void tearDown() {
if (driver != null) {
driver.quit();
}
}

Path getResource(String fileName) throws URISyntaxException {
URL refImgUrl = getClass().getClassLoader().getResource(fileName);
return Paths.get(refImgUrl.toURI()).toFile().toPath();
}

private String getResourceB64(String fileName) throws URISyntaxException, IOException {
Path refImgPath = getResource(fileName);
return Base64.getEncoder().encodeToString(Files.readAllBytes(refImgPath));
}

String getReferenceImageB64(String fileName) throws URISyntaxException, IOException {
return getResourceB64("images/" + fileName);
}

}

You can run this script or test directly to the main class.

Result Analysis 

After the first execution, We save the images in the baseline for the app’s homepage and after adding items to the cart. 

Result Analysis

  • We put a threshold value in the code for the comparison of images.
  • If the actual getScore value is greater than the threshold, then the test case getting failed. 
  • If the getScroe value is less than the threshold value then the test case getting pass. 
private final static double MATCH_THRESHOLD = 0.99;


if (res.getScore() < MATCH_THRESHOLD) {
File failViz = new File(Validation_path + "/FAIL_" + checkName + ".png");
res.storeVisualization(failViz);
throw new Exception(
String.format("Visual check of '%s' failed; similarity match was only %f, and below the threshold of %f. Visualization written to %s.",
checkName, res.getScore(), MATCH_THRESHOLD, failViz.getAbsolutePath()));}

// Otherwise, it passed!
System.out.println(String.format("Visual check of '%s' passed; similarity match was %f",
checkName, res.getScore()));

Image Comparison Commands

When you run this command with the appropriate image byte arrays (in this example, img1 is our baseline and img2 is the snapshot we want to investigate), and the right options object (it must have a version of SimilarityMatchingOptions), a SimilarityMatchingResult object is produced.

  • The most important function of the result object is getScore, which will tell us how comparable the two photographs we submitted were in terms of a score between 0 and 1. 
  • We just assess whether the score is greater than a predefined threshold to translate this into genuine validation (which should be selected experimentally based on your app). 
  • If so, we don’t believe there are any discernible changes. If not, we may issue an exception, causing the test to fail, or find another way to indicate the difference.
SimilarityMatchingResult res = driver.getImagesSimilarity(baselineImg, driver.getScreenshotAs(OutputType.FILE), opts);

Best Practices for Appium Visual Testing

  • For the tool to pass your testing, it must be able to handle anti-aliasing, pixel offsets, etc.
  • Release faster with DOM snapshotting and advanced parallelization capabilities designed for executing complex test suites at scale
  • Review, collaborate, and approve snapshots, keeping the rest of the team updated throughout the process
  • Choose a tool like Percy that is capable of ignoring false positives in visual testing.
  • Test automation should be capable of dealing with dynamic and shifting information.
  • Don’t rely on threshold settings or error ratios. The only thing that should be important is if a person can tell the difference and whether it will affect how the user interacts with the product.
  • The automation program must be able to evaluate the page’s structure and perform layout comparisons.
  • Validate the full UI page rather than individual parts. More comprehensive coverage will result from this. You risk missing unexpected issues if you validate only particular components.
  • Put the correct image data in the framework to ignore more thresholds.

Get Started with Automated Visual Testing

We’ve seen how to use Appium’s built-in image processing technology (powered by OpenCV) to build out some basic visual testing capabilities. In this article, we have complete function testing using visual testing. We perform small operations, but you can create a framework using this article. 

Appium Visual Testing: The Essential Guide

Tags
Appium Visual Testing

Featured Articles

How to report bugs in Appium UI Testing?

Strategies to Optimize Visual Testing

Curated for all your Testing Needs

Actionable Insights, Tips, & Tutorials delivered in your Inbox
By subscribing , you agree to our Privacy Policy.
thank you illustration

Thank you for Subscribing!

Expect a curated list of guides shortly.