All about TestNG Listeners in Selenium

Listeners are TestNG annotations that literally “listen” to the events in a script and modify TestNG behavior accordingly. These listeners are applied as interfaces in the code. For example, the most common usage of listeners occurs when taking a screenshot of a test that has failed and the reason for its failure. Listeners also help with logging and generating results.

This article will delve into the different types of listeners the popular framework TestNG provides.

What are TestNG Listeners?

TestNG listeners are interfaces in the TestNG framework that allow you to hook into the test execution process. They enable customization of test behavior, such as logging, reporting, or taking actions before, after, or when a test fails, passes, or is skipped.

Benefits of using TestNG Listeners with Selenium

TestNG Listeners are one of the key features of TestNG, and when used in conjunction with Selenium, they offer several benefits.

• Enhanced Test Reporting: By implementing listeners, you can capture and log events occurring during test execution, such as test case start, test case failure, test case success, etc.
• Test Result Analysis: Define custom actions to be taken when a test fails, such as capturing a screenshot, logging additional information, or sending a notification. This enables you to take immediate corrective actions and facilitates efficient debugging.
• Test Data Manipulation: Listeners provide hooks to modify or manipulate test data during runtime. You can use listeners to dynamically update test data, parameters, or configurations before or after each test case execution.
• Test Execution Control: Listeners allow you to define conditions and logic for executing or skipping tests based on specific criteria.
• Test Parallelization: TestNG parallel test execution and listeners play a crucial role in parallel test management. This can significantly reduce the test execution time, enabling quicker release cycles.
• Custom Test Execution Behaviors: The flexibility to define custom behaviors during test execution allows you to tailor the testing framework to suit your requirements and enhance test automation capabilities.

Types of TestNG Listeners in Selenium

Listeners are implemented in code via interfaces to modify TestNG behaviour. Listed below are the 8 most commonly used TestNG listeners:

These listeners can be implemented in TestNG in the following ways:

Using tag listener(<Listeners>) in a testNG.xml file

Using the listener annotation(@Listeners) in a testNG class as below:

@Listeners(com.Example.Listener.class)

Now, let’s dig into the TestNG listeners in detail.

1. IAnnotationTransformer

The best part about TestNG is the naming convention it uses for its keywords, like the ‘listener’ which listens to the code. Similarly, IAnnotationTransformer transforms the TestNG annotations at run time.

A scenario may appear in which the user seeks to override the content of the annotation based on a condition.

In such a case, making changes in the source code is unnecessary. Simply use IAnnotationTransformer to override the content of the annotations.

IAnnotationTransformer has only one method named transform() that accepts four parameters:

• ITestAnnotation annotation
• Class testClass
• Constructor testConstructor
• Method testMethod

Let’s implement this listener in code, to understand its usage. This scenario will change invocation count at run time for the required method of TestNG class.

import org.testng.annotations.Test;
public class IAnnotationTransformerWithExample {

MyListener obj=new MyListener();
@Test(invocationCount=5)
public void changeInvocationCountOfMethod()
{
System.out.println("This method have invocation count set to 5 but at run time it shall become "+ obj.counter);
}

}

The class implementing the interface IAnnotationTransformer that shall change this invocation count:

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;

public class MyListener implements IAnnotationTransformer {

int counter=3;

@Override
public void transform(ITestAnnotation testAnnotation, Class testClass, Constructor testConstrutor, Method testMethod)
{
if (testMethod.getName().equals("ChangeInvocationCountOfMethod")) {
System.out.println("Changing invocation for the following method: " + testMethod.getName());
testAnnotation.setInvocationCount(counter);

}

}
}

TestNG.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IAnnotationTransformerWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

Note: @Listener annotation can contain any class that extends ITestNGListener except the IAnnotationTransformer. The reason is that the latter listener needs to be informed at the earliest to TestNG so that they can override the annotation at runtime. Hence they should be mentioned in the testNG.xml file.

2. IExecutionListener

As the name suggests, it monitors the beginning and end of TestNG execution. This listener is mainly used to start/stop the server while starting or ending code execution. It may also inform respective stakeholders via email that execution shall start or when it ends. It has two methods:

• onExecutionStart() – invoked before TestNG starts executing the suites
• onExecutionFinish() – invoked after all TestNG suites have finished execution

Let’s look at an example. This example has a class with 5 methods which shall be executed after the onExecutionStart method of the IExecutionListener interface. After these methods are completed the onExecutionFinish method shall be executed. The two methods in this example highlight the start and end time of the test.

import org.testng.annotations.Test;

public class IExecutionListenerWithExample {

@Test
public void method1()
{
System.out.println("this method is method 1");

}

@Test
public void method2()
{
System.out.println("this method is method 2");

}

@Test
public void method3()
{
System.out.println("this method is method 3");

}

@Test
public void method4()
{
System.out.println("this method is method 4");

}

@Test
public void method5()
{
System.out.println("this method is method 5");

}

}

Class implementing IExecutionListener interface:

import java.sql.Time;

import org.testng.IExecutionListener;

public class MyListener implements IExecutionListener {

@Override
public void onExecutionFinish() {
long endTime= System.currentTimeMillis();
System.out.println("Inform all the suite have finished execution at"+ endTime);

}

@Override
public void onExecutionStart() {
long startTime= System.currentTimeMillis();
System.out.println("Inform all the suite have started execution at"+ startTime);

}

}

TestNG.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IExecutionListenerWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

3. IHookable

If a class implements this interface, its run method will be invoked instead of each test method. Using the IHookCallBack parameter’s callback method, the test method’s invocation can be performed.

It has a single method name run, which accepts two parameters.run(IHookCallBack callBack, ITestResult testResult)Now let’s look into its real-time example.

In this example, based on a certain parameter value, the test shall be skipped using the IHookable listener interface. These values will be provided by a data provider in a separate TestNG class.

import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class IHookableListenerWithExample {

@Test(dataProvider="parametersToBeSent")
public void t(String parameter) {
System.out.println("test method to be called with the following parameter is " + parameter);
}

@DataProvider
public Object[][] parametersToBeSent() {
return new Object[][]{{"parameter 1"}, {"parameter 2"}, {"parameter 3"}};
}
}

The class implementing the IHookable listener interface:

import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.ITestResult;

public class MyListener implements IHookable {

@Override
public void run(IHookCallBack callBack, ITestResult testResult) {

Object[] parameterValues = callBack.getParameters();
if (parameterValues[0].equals("parameter 3")) {
System.out.println("Skip the required parameter");
} else {
callBack.runTestMethod(testResult);
}

}

}

The testNG.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IHookableListenerWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console output:

4. IInvokedMethodListener

This listener gets invoked before and after a method in TestNG. These methods constitute both test and other configuration methods. These listeners help set up configuration or other cleanup activities. It contains two methods:

• beforeInvocation(): this method gets invoked before every method
• afterInvocation(): this method gets invoked after every method

Let’s look at an example. The TestNG class contains different configuration methods. The other class implements the InvokedMethodInterceptor which implements the beforeInvocation and afterInvocation methods. These defined methods execute before and after every config method of the TestNG class.

import org.testng.annotations.AfterMethod;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class IInvokedMethodListenerWithExample {

@BeforeSuite
public void method1() {
System.out.println("before suite");
}

@BeforeMethod
public void method2() {
System.out.println("before method");
}

@Test
public void method3() {
System.out.println("test method 1 ");
}

@Test
public void method4() {
System.out.println("test method 2 ");
}

@AfterMethod
public void method5() {
System.out.println("after method");
}

@AfterSuite
public void afterSuite() {
System.out.println("after suite");
}
}

The class implementing the InvokedMethodListener interface:

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
public class MyListener implements IInvokedMethodListener {

@Override
public void afterInvocation(IInvokedMethod method, ITestResult result) {
System.out.println("This method is invoked after every config method - " + method.getTestMethod().getMethodName());

}

@Override
public void beforeInvocation(IInvokedMethod method, ITestResult result) {
System.out.println("This method is invoked before every config method - " + method.getTestMethod().getMethodName());

}

}

TestNG.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IInvokedMethodListenerWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

5. IMethodInterceptor

This listener helps to alter the methods that TestNG is supposed to run. It gets invoked just before TestNG invokes the methods. It just has one method, intercept, that returns an altered list of methods. Let’s look at an example.

The code here will run only methods with priority 1 in the test class. Other methods with different priorities shall not be executed. This will be done after implementing the IMethodInterceptor listener.

import org.testng.annotations.Test;

public class IMethodInterceptorWithExample {

@Test(priority=2)
public void method1() {
System.out.println("Method 1 will not be executed");
}

@Test(priority=2)
public void method2() {
System.out.println("Method 2 will not be executed");
}

@Test(priority=1)
public void method3() {
System.out.println("Method 3 will be executed");
}

@Test(priority=1)
public void method4() {
System.out.println("Method 4 will be executed");
}
}

The IMethodInterceptor class implementing the interface:

import java.util.ArrayList;
import java.util.List;

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;
import org.testng.annotations.Test;

public class MyListener implements IMethodInterceptor {

@Override
public List<IMethodInstance> intercept(List<IMethodInstance> methodsInstance, ITestContext testContext) {
List<IMethodInstance> result = new ArrayList<IMethodInstance>();
for (IMethodInstance method : methodsInstance) {
Test testMethod = method.getMethod().getConstructorOrMethod().getMethod().getAnnotation(Test.class);
if (testMethod.priority() == 1) {
result.add(method);
}
}
return result;
}

}

TestNG.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IMethodInterceptorWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

6. IReporter

This listener helps to generate custom reports in TestNG based on desired conditions. It contains a method called generateReport() which is invoked when all suites of TestNG are executed. The technique uses three arguments:

• xmlSuite: It includes a list of suites for execution in the XML file
• suites: It contains all information about test execution and suites like class name, package name, method name, and test execution results
• outputDirectory: It has the path where the report shall be saved.

Let’s look at an example of how to customize reports through IReporter listeners. In this example, through the IReporter listener, the code shall run only methods belonging to a particular group. In this class, the group has been defined as ‘Sanity’, which shall be executed.

The processes which are not part of the group shall not be executed. In the other class that implements IReporter, the generateReport() method has been used to customize the results accordingly. The customized results shall be visible in the console with a corresponding report generated under the suite folder name specified in the testNG.xml file

import org.testng.Assert;
import org.testng.annotations.Test;

public class IReporterWithExample {

@Test(groups="smoke")
public void testcase1() {
System.out.println("This test case will pass");
}

@Test(groups="smoke")
public void testcase2() {
System.out.println("This test case will fail");
Assert.assertTrue(false);
}

@Test
public void testcase3() {
System.out.println("this tet case does not belong to the group smoke");
}

}

The class implementing the IReporter listener interface:

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.testng.IReporter;
import org.testng.IResultMap;
import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;
import org.testng.xml.XmlSuite;

public class MyListener implements IReporter {

@Override
public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
// TODO Auto-generated method stub
ISuite suite = suites.get(0);
Map<String, Collection<ITestNGMethod>> methodsByGroup = suite.getMethodsByGroups();
Map<String, ISuiteResult> tests = suite.getResults();
for (String key : tests.keySet()) {
System.out.println("Key: " + key + ", Value: " + tests.get(key));
}
Collection<ISuiteResult> suiteResults = tests.values();
ISuiteResult suiteResult = suiteResults.iterator().next();
ITestContext testContext = suiteResult.getTestContext();
Collection<ITestNGMethod> perfMethods = methodsByGroup.get("smoke");
IResultMap failedTests = testContext.getFailedTests();
for (ITestNGMethod perfMethod : perfMethods) {
Set<ITestResult> testResultSet = failedTests.getResults(perfMethod);
for (ITestResult testResult : testResultSet) {
System.out.println("Test " + testResult.getName() + " failed, error " + testResult.getThrowable());
}
}
IResultMap passedTests = testContext.getPassedTests();
for (ITestNGMethod perfMethod : perfMethods) {
Set<ITestResult> testResultSet = passedTests.getResults(perfMethod);
for (ITestResult testResult : testResultSet) {
System.out.println("Test " + testResult.getName() + " passed, time took " +
(testResult.getEndMillis() - testResult.getStartMillis()));
}
}

}

}

Below is the tesNG.xml for the classes to be run:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Report_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IReporterWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

A corresponding customized report shall also be created under the folder name, the suite name defined in the xml file. In this example, it is Report_Suite.

Sample Report:

7. ISuiteListener

As the name suggests, this listener works at the suite level. It listens and runs before the start and end of suite execution. It contains two methods:

• onStart: invoked before test suite execution starts
• onFinish: invoked after test suite execution finishes.

The following example incorporates the ISuiteListener. The code will use two child classes containing before and after suites. These shall be run through a TestNG xml file containing a reference to the ISuiteListener class, which shall run its onStart and onFinish methods first and last respectively.

import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class MyListener implements ISuiteListener {

@Override
public void onFinish(ISuite suite1) {
System.out.println("onFinish function started of ISuiteListener " );

}

@Override
public void onStart(ISuite suite2) {
System.out.println("onStart function started of ISuiteListener " );

}

}

Below are the two respective child classes:

import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class ISuiteListenerWithExample {

@BeforeSuite
public void bsuite()
{
System.out.println("BeforeSuite method started for the first IsuiteListener example class");
}

@Test
public void test()
{
System.out.println("Test method started for the first IsuiteListener example class");
}

@AfterSuite
public void asuite()
{
System.out.println("AfterSuite method started for the first IsuiteListener example class");
}

}

And the second class:

import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class ISuiteListenerExample2 {

@BeforeSuite
public void bsuite()
{
System.out.println("BeforeSuite method started for the first IsuiteListener example 2 class");
}

@Test
public void test()
{
System.out.println("Test method started for the first IsuiteListener example 2 class");
}

@AfterSuite
public void asuite()
{
System.out.println("AfterSuite method started for the first IsuiteListener example 2 class");
}

}

Their respective xml files:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="SuiteOne">
<test thread-count="5" name="Test">
<classes>
<class name="ISuiteListenerExample2"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite2">
<test thread-count="5" name="Test">
<classes>
<class name="ISuiteListenerWithExample"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->

The final testNG xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<suite-files>
<suite-file path="ISuiteListenerWithExample.xml"> </suite-file>
<suite-file path="ISuiteListenerExample2.xml"> </suite-file>
</suite-files>
</suite>
<!-- Suite -->

Console Output:

8. ITestListener

This is the most frequently used TestNG listener. ITestListener is an interface implemented in the class, and that class overrides the ITestListener-defined methods. The ITestListener listens to the desired events and executes the methods accordingly.

It contains the following ways:

• onStart(): invoked after test class is instantiated and before execution of any testNG method.
• onTestSuccess(): invoked on the success of a test
• onTestFailure(): invoked on the failure of a test
• onTestSkipped(): invoked when a test is skipped
• onTestFailedButWithinSuccessPercentage(): invoked whenever a method fails but within the defined success percentage
• onFinish(): invoked after all tests of a class are executedThe above-mentioned methods use the parameters ITestContext and ITestResult. The ITestContext is a class that contains information about the test run. The ITestResult is an interface that defines the result of the test.

Now, let’s look at an example showcasing the use of this listener.

Below is a listener class that implements ITestListener:

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class MyListener implements ITestListener {

@Override
public void onFinish(ITestContext contextFinish) {
System.out.println("onFinish method finished");

}

@Override
public void onStart(ITestContext contextStart) {
System.out.println("onStart method started");
}

@Override
public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
System.out.println("Method failed with certain success percentage"+ result.getName());

}

@Override
public void onTestFailure(ITestResult result) {
System.out.println("Method failed"+ result.getName());

}

@Override
public void onTestSkipped(ITestResult result) {
System.out.println("Method skipped"+ result.getName());

}

@Override
public void onTestStart(ITestResult result) {
System.out.println("Method started"+ result.getName());

}

@Override
public void onTestSuccess(ITestResult result) {
System.out.println("Method passed"+ result.getName());

}

}

The class below contains four methods, showcasing one method being passed, one method being failed, one method being skipped and one method being passed with a defined success percentage:

import org.testng.Assert;
import org.testng.SkipException;
import org.testng.annotations.Test;
import org.testng.asserts.SoftAssert;

public class ItestListenerWithExample {

int i=0;

@Test
public void testMethod1()
{
System.out.println("This method will pass and will invoke the onTestSuccess method of ITestlistener");
int i=10;
Assert.assertEquals(i, 10);
}

@Test
public void testMethod2()
{
System.out.println("This method will fail and will invoke the onTestFailure method of ITestlistener");
int i=10;
Assert.assertEquals(i, 11);
}

@Test
public void testMethod3()
{
System.out.println("This method will skip and will invoke the onTestSkipped method of ITestlistener");
throw new SkipException("Skipping this test case.");

}

@Test(successPercentage=50, invocationCount=5)
public void testMethod4()
{
i++;
System.out.println("Test Failed But Within Success Percentage Test Method, invocation count: " + i);
if (i == 1 || i == 2) {
System.out.println("this will be Failed");
Assert.assertEquals(i, 100);
}
}
}

Below is the testNG xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="Listeners_program">
<classes>
<class name="ItestListenerWithExample"></class>
</classes>
</test>
</suite>
<!-- Suite -->

Console output:

Talk to an Expert

Why use BrowserStack Automate to run TestNG Selenium Tests?

You should run Selenium TestNG Tests on a real device cloud like BrowserStack Automate for below reasons:

• Realistic Testing Conditions: Real device clouds provide access to a broad spectrum of devices and environments, ensuring tests reflect actual user conditions accurately.
• Enhanced Security: Maintained with high security standards, real device clouds offer secure, isolated testing environments, minimizing data breach risks.
• Broad Browser and OS Coverage: Helps identify compatibility issues across various browsers and operating systems, enhancing user experience.
• Performance Insights: Real devices yield authentic performance data essential for optimizing application responsiveness.
• Scalability and Accessibility: Facilitates scalable and accessible testing, suitable for distributed teams.
• CI/CD Integration: Integrates smoothly with CI/CD pipelines for continuous testing and early issue detection.
• Cost-Effectiveness: Although initially more costly, it saves on long-term expenses related to fixes and support.

Conclusion

• Listeners play an essential role when designing an end-to-end framework for web applications. They make execution smoother, with required actions performed after events are triggered.
• From taking screenshots when test cases fail, to customizing reports to changing values during run time, TestNG Listeners help to refine and make automation testing easier.

With Parallel Testing on a Cloud Selenium Grid, BrowserStack Automate allows you to run multiple tests in parallel across various browsers/devices and OS combinations. Run hundreds of tests concurrently to speed up the execution time of your test suite by more than 10x.

Run Selenium Tests on Real Devices