How to perform Test Automation with CircleCI

CircleCI is a Continuous Integration and Delivery platform that helps teams perform build, test and release seamlessly and quickly. It provides integrations into multiple tools and can be configured to run complex pipelines. 

CircleCI reduces the overhead of having a dedicated server as it is cloud-based. The enterprise version is also low on maintenance. The cloud-based platform offers credit-based plans that are scalable and help in deploying applications faster.

Why integrate your Test Automation Suite with CircleCI?

In most organizations, there would be a bunch of Test Automation Engineers working on the same project. When there are multiple additions to the automation code, it is a good practice to build for every commit, to check what impact that commit had on the codebase. To do this manually for every commit is a mundane task. At times, the person who wrote the automation code may forget to trigger the build manually and might have committed a code containing a bug. This will cause all the further tests to fail. 

By integrating the Test Automation repositories with orchestration tools like CircleCI, each commit to the main branch, and every pull request raised against the main branch gets built and the test results are also integrated within the Github UI. This helps the teams have more confidence in the test suite they have built as it would highlight the issues early in the development phase.  

From a continuous integration perspective, testing is an important phase in the pipeline and it would assess the accuracy of the code early in the development cycle. Any failures are reported to the developers immediately and a fix can be provided so that it will not grow into a bigger issue later on. 

How to perform Test Automation with CircleCI

Step 1: Sample Test Automation Code

Let’s create a sample test automation code to later integrate with CircleCI. For the purpose of this demo, here creating a simple test case that opens a website using TestNG. 

Below are the contents of the test class, pom.xml, and testng.xml files. 

NewTest.java

package tests;

import org.openqa.selenium.chrome.ChromeDriver;

import org.testng.annotations.Test;

import org.testng.annotations.AfterTest;

import org.testng.annotations.BeforeTest;


public class NewTest {

public ChromeDriver driver;

@BeforeTest

public void beforeClass() {

 System.setProperty("webdriver.chrome.driver", "chromedriver.exe");

     driver = new ChromeDriver();

}

    @Test

    public void openWebsite() {

     driver.get("https://www.google.com/");

    }

    @AfterTest

    public void afterClass() {

    driver.close();

  }

}

pom.xml 

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

  <modelVersion>4.0.0</modelVersion>

  <groupId>mavenPackage</groupId>

  <artifactId>MavenProject</artifactId>

  <version>0.0.1-SNAPSHOT</version>


  <dependencies>


<!-- Selenium -->

<dependency>

  <groupId>org.seleniumhq.selenium</groupId>

  <artifactId>selenium-java</artifactId>

  <version>4.2.0</version>

</dependency>

<!-- TestNG -->

<dependency>

  <groupId>org.testng</groupId>

  <artifactId>testng</artifactId>

  <version>7.4.0</version>

  <scope>test</scope>

</dependency>


  </dependencies>


  <build>

   <plugins>

   <plugin>

   <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-compiler-plugin</artifactId>

            <version>3.10.1</version>

            <configuration>

            <source>1.8</source>

            <target>1.8</target>

            </configuration>

   </plugin>


   <plugin>

   <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-surefire-plugin</artifactId>

            <version>2.20</version>

            <configuration>

            <suiteXmlFiles>

            <suiteXmlFile>testng.xml</suiteXmlFile>

            </suiteXmlFiles>

            </configuration>

   </plugin>

   </plugins>  

  </build>

  

</project>

testng.xml

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Suite" parallel="false">

  <test name="Test">

    <classes>

      <class name="tests.NewTest"/>

    </classes>

  </test> <!-- Test -->

</suite> <!-- Suite -->

Step 2: Create a free account on CircleCI

CircleCI offers a cloud build platform wherein we can easily get started by creating a free account. Please note the limitations of the free tier. Click on the Start Building for Free button. 

This will take you to a page where you will need to select the account using which you will create your CircleCI account. For ease of setup, you can choose to integrate your GitHub account.

Click on Sign up with GitHub and it will link to your GitHub organization if it is already logged in. Else login to your Github organization when prompted. Once this is done, Github asks for authorization to allow access to CircleCI to integrate with your account. Click on Authorize CircleCI

Step 3: Select your organization

Once the login is completed, you will be asked to select your GitHub organization. Select the one you want to link with CircleCI.

Once done, the homepage of CircleCI opens. This will have your repositories populated. The left pane offers buttons for Dashboards, Projects, Organization Settings, and Plan. 

For this demo, let’s use the CircleCI_Selenium_Demo repository. 

Step 4: Create a Project 

Click on Set Up Project for the repository that holds your test automation code. This will open a dialogue box with the following options.

Select the default branch from which you want to run the project. CircleCI relies on a config.yml file that holds all the configuration of the pipeline. Here adding a sample config.yml file to this project by choosing the Hello World template after selecting the Fast option. 

If you already are familiar with creating a config.yml, you can create one in your test automation repo. Below is the config.yml that I have used for this project. 

config.yml

version: 2

jobs:

  build:

    docker:

      # specify the version you desire here

      - image: circleci/openjdk:8-jdk

    working_directory: ~/circleCI_Selenium_Demo

    environment:

      # Customize the JVM maximum heap limit

      MAVEN_OPTS: -Xmx3200m  

    steps:

      - checkout

      # Download and cache dependencies


      - restore_cache:

          keys:

          - v1-dependencies-{{ checksum "pom.xml" }}

          # fallback to using the latest cache if no exact match is found

          - v1-dependencies-


      - run: mvn dependency:go-offline


      - run:

          name: Install latest version of ChromeDriver Sample

          command: |

            sudo wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -

            sudo sudo sh -c 'echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'

            sudo apt-get update

            sudo apt-get install google-chrome-unstable


      - run:

          name: Install latest version of ChromeDriver Sample

          command: |

            sudo wget https://chromedriver.storage.googleapis.com/2.31/chromedriver_linux64.zip

            sudo unzip -o chromedriver_linux64.zip

            sudo rm chromedriver_linux64.zip

            sudo mv chromedriver /usr/bin/

            sudo chmod 777 /usr/bin/chromedriver

            sudo apt-get install libxi6 libgconf-2-4

            sudo apt-get -y install xvfb gtk2-engines-pixbuf

            sudo apt-get -y install xfonts-cyrillic xfonts-100dpi xfonts-75dpi xfonts-base xfonts-scalable

            sudo apt-get install xvfb

            sudo apt-get -y install imagemagick x11-apps
   

      - run:

          name: Running X virtual framebuffer

          command: Xvfb :0 -ac &


      - run:

          name: Run Tests

          command: |

            export DISPLAY=:99


      - save_cache:

          paths:

            - ~/.m2

          key: v1-dependencies-{{ checksum "pom.xml" }}
      

      # run tests!

      - run: mvn clean test


      - store_artifacts:

          path: target/surefire-reports

          destination: tr1

      - store_test_results:

          path: target/surefire-reports

Note that upon every commit, CircleCI will trigger a new run of the pipeline automatically. Also, when you use the Hello World template for creating a sample config.yml file, you would have noticed that it would automatically create a new branch in GitHub circleci-project-setup. 

You can continue to edit your config.yml file in the branch and merge it back onto the main branch when needed. The CircleCI project can be configured to run the tests from any branch on the repo.

Run CircleCI Selenium Tests on Real Devices

It can be noted that for every PR raised on Github, a new run is triggered on CircleCI and the status of the run would get updated in GitHub. This serves as an additional check while merging PRs.

This is how we can configure and run our test automation using CircleCI. 

Integrating Browserstack with CircleCI

CircleCI relies on config.yml for all the configurations and integrations. To run tests on real devices, you can add integrations for Browserstack as well in the config.yml file. This allows access to 3000+ real device browser combinations for wide test coverage and supports popular test automation frameworks like Selenium, Cypress, Playwright, and Puppeteer. 

There are some variables like grid config, username, and access key, that we will need to set in order to get the integration to work. 

Configuration of circle.yml file for Selenium Tests

machine:

  environment:

    BROWSERSTACK_USERNAME: “<user-name>”

    BROWSERSTACK_ACCESS_KEY: "<access-key>j"

    BROWSERSTACK_LOCAL: false

    BROWSERSTACK_LOCAL_IDENTIFIER: "identifier"

CircleCI environment variables in your code can be seen as below:

String username = System.getenv("BROWSERSTACK_USERNAME");

String accessKey = System.getenv("BROWSERSTACK_ACCESS_KEY");

String browserstackLocal = System.getenv("BROWSERSTACK_LOCAL");

String browserstackLocalIdentifier = System.getenv("BROWSERSTACK_LOCAL_IDENTIFIER");

DesiredCapabilities capabilities = new DesiredCapabilities();

capabilities.setCapability("os", "Windows");

capabilities.setCapability("browser", "chrome");

capabilities.setCapability("browserstack.local", browserstackLocal);

capabilities.setCapability("browserstack.localIdentifier", browserstackLocalIdentifier);

driver = new RemoteWebDriver(new URL("https://" + username + ":" + accessKey + "@hub.browserstack.com/wd/hub"), capabilities);

No matter which testing framework you choose, it is always suggested to test on a real device cloud under real user conditions for better accuracy of test results. Test on different device browser combinations simultaneously using Cloud Selenium Grid that leverages Parallel Testing for a faster test cycle, and wider test coverage.

Try BrowserStack for Free