ChatBot Testing : A Beginner’s Guide

What is a Chatbot?

A chatbot is an artificial intelligence (AI) program that can initiate and chat with a user in natural language via messaging. For example, the chatbot on Domino’s website allows users to get what they want with a few clicks. It provides a set of options the users can choose from, with an option for interaction with a human as well.

Chatbots have become an essential part of online businesses because they allow for performing previously impossible tasks: helping customers 24 hours a day, resolving queries without the need for human intervention, and supporting multiple customers at the same time. Other important advantages that chatbots provide for businesses include:

• High-speed resolution of queries
• Reduced operational costs
• Reduced human error
• 24/7 multi-user support 
• Automation of certain tasks

Evolution of Chatbots

The concept behind chatbots existed long before the computer was introduced. In fact, the first chatbot was introduced before the launch of personal computers. It was developed in the MIT Artificial Intelligence Laboratory by Joseph Weizenbaum in 1966 and was named Eliza. The bird’s eye view of chatbot evolution is shown below. With the advent of AI, the trend continues to grow with newer features, sharper capabilities, and more “human” responses.

How do businesses benefit from Chatbots?

When it comes to building strong client relationships, chatbots are a beneficial tool. By engaging and communicating with website visitors, companies can use chatbots to provide consistent interaction with customers.

By integrating chatbots, they can not only achieve marketing goals but also drive sales and improve customer service. Chatbots can positively impact customer communication and drive business growth in multiple ways: 

1. Increase customer engagement and meet customer service expectations 
2. Improve lead generation
3. Reduce customer service costs since only major issues will require human intervention
4. Easier to monitor customer behavior for insights 
5. Streamline customer onboarding and responses

What is Chatbot Testing

The majority of chatbots do not comprehend user intent, and many of them are not verified via adequate testing. When they cannot understand human voice or instructions they fail. 

Customers must find chatbots useful in order to make them successful. Customer intent is what the customer wants to get done, or the ultimate outcome they’re seeking when they engage with a chatbot.

Therefore, it is very essential that chatbots should be programmed and tested with multiple input values and formats to make them effective.

Chatbots need to be programmed to understand customer intent. Because of the wildly different ways in which humans communicate individual intent, it takes significant data to represent the input values that chatbots must be able to process. 

This training data is a set of scenarios expressing different kinds of intent. Chatbots use these data points to create mathematical models for recognizing said intent. It must be then tested to ensure that its response is as expected in every scenario.

The image below depicts a step-wise approach to chatbot testing. It starts with identifying the use case for the test and continues till testing non-functional aspects like reliability, availability, maintenance, scalability, usability, etc.

Chatbots can be tested via a number of techniques – using RPA, Security testing, UFT testing, and many more. Additionally, there are various tools like Botium useful for testing chatbots.

For this piece, let’s explore how to handle the chatbot using Selenium.

Chatbot Automation Testing using Selenium

This example will use the chatbot on the Dominos website. The main task of the automation script is to locate the virtual assistant, click on it, and then select the stores near me option.

import org.openqa.selenium.By;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.support.ui.ExpectedConditions;

import org.openqa.selenium.support.ui.WebDriverWait;

public class Homepage {

public static void main(String[] args) {

System.setProperty("webdriver.chrome.driver","C:\Path to the driver");

WebDriver driver = new ChromeDriver();

String baseUrl = "”Domino’s Pizza – Order Online | Get 2 Regular Pizza @99 Each (dominos.co.in);

driver.get(baseUrl);

driver.findElement(By.xpath("////body/div[@id='ymPluginDivContainerInitial']/div[@id='ymDivCircle']/img[1]")).click();

WebDriverWait wait = new WebDriverWait(driver,60) ;

wait.until(ExpectedConditions.presenceOfElementLocated(By.xpath("//strong[contains(text(),'Stores near me')]")));

driver.findElement(By.xpath("//strong[contains(text(),'Stores near me')]")).click();

System.out.println("Found the button, clicked on that...");

driver.close();

System.exit(0);

}

}

That’s how it works. The output will be as shown below.

Now, the same test can be executed using Browserstack automate as shown below. 

Chatbot Testing using Browserstack Automate

BrowserStack is a cloud-based website and mobile testing platform that offers 3000+ real browsers and devices (desktop and mobile) for an instant, on-demand testing. 

• Login to BrowserStack. Launch Automate If you don’t have an account already, sign up for free. 
• The sample code is already available. Just modify and add your code.
• Once it is configured, set up the dependencies and desired capabilities to run the test.
• Follow the detailed instructions below if you are using Gradle or Maven to build your Java project. Add the maven dependency to your pom.xml file or you can directly configure your build path by adding the selenium jar file to your library in order to run the test case.
  

Next, configure the browser and device combinations as shown below.

As a next step, copy the code snippet, do the necessary changes and run it on the IDE. In the below example, you can test the code on multiple browsers and devices. If you are new to BrowserStack automate, pause here, refer to the documentation on Browserstack Automate and then execute the below test.

import java.net.MalformedURLException;
import java.net.URL;
import java.util.Hashtable;
import java.util.Iterator;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import org.openqa.selenium.By;
import org.openqa.selenium.JavascriptExecutor;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

class TestClass1 implements Runnable {
public void run() {

Hashtable<String, String> capsHashtable = new Hashtable<String, String>();
 capsHashtable.put("browser", "chrome");
 capsHashtable.put("browser_version", "93.0");
 capsHashtable.put("os", "Windows");
 capsHashtable.put("os_version", "10");
                 capsHashtable.put("build", "browserstack-build-1");

capsHashtable.put("name", "Thread 1");
              mainTestClass r1 = new mainTestClass();
                       r1.executeTest(capsHashtable);
  }
}
class TestClass2 implements Runnable {
public void run() {

Hashtable<String, String> capsHashtable = new Hashtable<String, String>();
 capsHashtable.put("device", "OnePlus 7");
 capsHashtable.put("os_version", "9.0");
 capsHashtable.put("browserName", "android");
 capsHashtable.put("realMobile", "true");
 capsHashtable.put("build", "browserstack-build-1");
 capsHashtable.put("name", "Thread 2");
 mainTestClass r2 = new mainTestClass();
               r2.executeTest(capsHashtable);
 }
}
class TestClass3 implements Runnable {
 public void run() {

Hashtable<String, String> capsHashtable = new Hashtable<String, String>();
 capsHashtable.put("browser", "safari");
 capsHashtable.put("browser_version", "latest");
 capsHashtable.put("os", "OS X");
 capsHashtable.put("os_version", "Big Sur");
 capsHashtable.put("build", "browserstack-build-1");
 capsHashtable.put("name", "Thread 3");
 mainTestClass r3 = new mainTestClass();
              r3.executeTest(capsHashtable);
   }
}

public class mainTestClass {

public static final String USERNAME = "";
public static final String AUTOMATE_KEY = "";
public static final String URL = "https://" + USERNAME + ":" + AUTOMATE_KEY + "@hub-cloud.browserstack.com/wd/hub";
 public static void main(String[] args) throws Exception {
 Thread object1 = new Thread(new TestClass1());
 object1.start();
 Thread object2 = new Thread(new TestClass2());
 object2.start();
 Thread object3 = new Thread(new TestClass3());
 object3.start();
   }
 public void executeTest(Hashtable<String, String> capsHashtable) {

String key;
                        DesiredCapabilities caps = new DesiredCapabilities();
 // Iterate over the hashtable and set the capabilities
                        Set<String> keys = capsHashtable.keySet();
                        Iterator<String> itr = keys.iterator();
                         while (itr.hasNext()) {
                         key = itr.next();
                         caps.setCapability(key, capsHashtable.get(key));
}

     WebDriver driver;
                     try {

driver = new RemoteWebDriver(new URL(URL), caps);
                                   JavascriptExecutor jse = (JavascriptExecutor)driver;
                                  // Clicking on the Domino's Website

driver.get("https://www.dominos.co.in/");
     driver.findElement(By.xpath("////body/div[@id='ymPluginDivContainerInitial']/div[@id='ymDivCircle']/img[1]")).click();

// Setting the status of test as 'passed' or 'failed' based on the condition; 
WebDriverWait wait = new WebDriverWait(driver, 5);
try {
 WebDriverWait wait = new WebDriverWait(driver,60) ;
wait.until(ExpectedConditions.presenceOfElementLocated(By.xpath("//strong[contains(text(),'Stores near me')]")));
driver.findElement(By.xpath("//strong[contains(text(),'Stores near me')]")).click();
System.out.println("Found the button, clicked on that...");
jse.executeScript("browserstack_executor: {\"action\": \"setSessionStatus\", \"arguments\": {\"status\": \"passed\", \"reason\": \"Title matched!\"}}")
 }
      catch(Exception e) {
jse.executeScript("browserstack_executor: {\"action\": \"setSessionStatus\", \"arguments\": {\"status\":\"failed\", \"reason\": \"Title not matched\"}}");
}
System.out.println(driver.getTitle());
                        driver.quit();
                        } catch (MalformedURLException e) {
                          e.printStackTrace();
            }
    }
}

Try Chatbot Testing using Selenium for Free

Now, on executing the test case, you will get the same output as you got above and multiple test cases will run on the BrowserStack automate dashboard as shown below.

Like any other software test, chatbot tests are best executed on real browsers, devices, and operating systems. Testing in real user conditions allows the tester to see exactly how the chatbot will behave when being accessed by actual customers. If testers do not have access to an in-house device lab, they can simply use a real device cloud to run the tests.

BrowserStack offers 3000+ real browsers and devices for manual and automation testing. QAs can test chatbots on websites as well as Android and iOS apps on thousands of real mobile devices, both latest and older ones. They can integrate seamlessly with the BrowserStack cloud via numerous testing frameworks – Appium, Espresso, XCUITest, or EarlGrey.

Testers can test chatbots in order to verify their compatibility with multiple native device features – geolocation testing, push notifications, preloaded images, network simulation, in-app purchase, time zones, languages, etc.

Simply sign up for free, select the device-operating combination required and start testing. The entire process is built to provide convenience, efficiency, speed, and accuracy to testers and developers alike. With a wide range of integrations and debugging tools, BrowserStack is built to not just identify bugs, but resolve them at the quickest.