All About TestNG Listeners

Listeners are TestNG annotations that literally “listen” to the events in a script and modify TestNG behaviour accordingly. These listeners are applied as interfaces in the code. For example, the most common usage of listeners occurs when taking a screenshot of a particular test that has failed along with the reason for its failure. Listeners also help with logging and generating results.
This article will delve into the different types of listeners provided by the popular framework TestNG.

Types of TestNG Listeners

Listeners are implemented in code via interfaces to modify TestNG behaviour. Listed below are the most commonly used TestNG listeners:

• IAnnotationTransformer
• IExecutionListener
• IHookable
• IInvokedMethodListener
• IMethodInterceptor
• IReporter
• ISuiteListener
• ITestListener

These listeners can be implemented in TestNG in the following ways:

• Using tag listener(<Listeners>) in a testNG.xml file
• Using the listener annotation(@Listeners) in a testNG class as below:    @Listeners(com.Example.Listener.class)

Now, let’s dig into the TestNG listeners in detail.

• IAnnotationTransformer: The best part about TestNG is the naming convention it uses for its keywords, like the ‘listener’ which listens to the code. Similarly, IAnnotationTransformer transforms the TestNG annotations at run time. A scenario may appear in which the user seeks to override the content of the annotation based on a condition. In such a case it is not necessary to make changes in the source code. Simply use IAnnotationTransformer to override the content of the annotations. IAnnotationTransformer has only one method named transform() that accepts four parameters:

1. 1. ITestAnnotation annotation
   2. Class testClass
   3. Constructor testConstructor
   4. Method testMethod

Let’s implement this listener in code, to understand its usage. This scenario will change invocation count at run time for the required method of TestNG class.

import org.testng.annotations.Test;
public class IAnnotationTransformerWithExample {

MyListener obj=new MyListener();
@Test(invocationCount=5)
public void changeInvocationCountOfMethod()
{
System.out.println("This method have invocation count set to 5 but at run time it shall become "+ obj.counter);
}

}

The class implementing the interface IAnnotationTransformer that shall change this invocation count:

import java.lang.reflect.Constructor;
import java.lang.reflect.Method;

import org.testng.IAnnotationTransformer;
import org.testng.annotations.ITestAnnotation;

public class MyListener implements IAnnotationTransformer {

int counter=3;

@Override
public void transform(ITestAnnotation testAnnotation, Class testClass, Constructor testConstrutor, Method testMethod)
{
if (testMethod.getName().equals("ChangeInvocationCountOfMethod")) {
System.out.println("Changing invocation for the following method: " + testMethod.getName());
testAnnotation.setInvocationCount(counter);

}

}
}

TestNG.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IAnnotationTransformerWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

• IExecutionListener: As the name suggests, it monitors the beginning and end of TestNG execution. This listener is mostly used to start/stop the server while starting or ending code execution. It may also be used to inform respective stakeholders via email that execution shall start or when it ends. It has two methods:

1. 1. onExecutionStart() – invoked before TestNG starts executing the suites
   2. onExecutionFinish() – invoked after all TestNG suites have finished execution

Let’s look at an example. This example has a class with 5 methods which shall be executed after the onExecutionStart method of the IExecutionListener interface. After these methods are completed the onExecutionFinish method shall be executed. The two methods in this example highlight the start and end time of the test.

import org.testng.annotations.Test;

public class IExecutionListenerWithExample {

@Test
public void method1()
{
System.out.println("this method is method 1");

}

@Test
public void method2()
{
System.out.println("this method is method 2");

}

@Test
public void method3()
{
System.out.println("this method is method 3");

}

@Test
public void method4()
{
System.out.println("this method is method 4");

}

@Test
public void method5()
{
System.out.println("this method is method 5");

}

}

Class implementing IExecutionListener interface:

import java.sql.Time;

import org.testng.IExecutionListener;

public class MyListener implements IExecutionListener {

@Override
public void onExecutionFinish() {
long endTime= System.currentTimeMillis();
System.out.println("Inform all the suite have finished execution at"+ endTime);

}

@Override
public void onExecutionStart() {
long startTime= System.currentTimeMillis();
System.out.println("Inform all the suite have started execution at"+ startTime);

}

}

TestNG.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IExecutionListenerWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

• IHookable: If a class implements this interface, its run method will be invoked instead of each test method. Using the callback method of the IHookCallBack parameter, the invocation of the test method can be performed. It has a single method name run, which accepts two parameters.run(IHookCallBack callBack, ITestResult testResult)Now let’s look into its real-time example. In this example, based on a certain parameter value, the test shall be skipped using the IHookable listener interface. These values will be provided by a data provider in a separate TestNG class.

  import org.testng.annotations.DataProvider;
  import org.testng.annotations.Test;
  
  public class IHookableListenerWithExample {
  
  @Test(dataProvider="parametersToBeSent")
  public void t(String parameter) {
  System.out.println("test method to be called with the following parameter is " + parameter);
  }
  
  @DataProvider
  public Object[][] parametersToBeSent() {
  return new Object[][]{{"parameter 1"}, {"parameter 2"}, {"parameter 3"}};
  }
  }

  The class implementing the IHookable listener interface:

  import org.testng.IHookCallBack;
  import org.testng.IHookable;
  import org.testng.ITestResult;
  
  public class MyListener implements IHookable {
  
  @Override
  public void run(IHookCallBack callBack, ITestResult testResult) {
  
  Object[] parameterValues = callBack.getParameters();
  if (parameterValues[0].equals("parameter 3")) {
  System.out.println("Skip the required parameter");
  } else {
  callBack.runTestMethod(testResult);
  }
  
  }
  
  }

  The testNG.xml file:

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
  <suite name="Parent_Suite">
  <listeners>
  <listener class-name="MyListener"/>
  </listeners>
  <test name="ItestReporter">
  <classes>
  <class name="IHookableListenerWithExample" />
  </classes>
  </test>
  </suite>
  <!-- Suite -->

  Console output:

  

   

• IInvokedMethodListener: This listener gets invoked before and after a method in TestNG. These methods constitute both test and other configuration methods. These listeners are useful for setting up configuration or other cleanup activities. It contains two methods:

1. 1. beforeInvocation(): this method gets invoked before every method
   2. afterInvocation(): this method gets invoked after every method

Let’s look at an example. The TestNG class contains different configuration methods. The other class implements the InvokedMethodInterceptor which implements the beforeInvocation and afterInvocation methods. These defined methods execute before and after every config method of the TestNG class.

import org.testng.annotations.AfterMethod;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class IInvokedMethodListenerWithExample {

@BeforeSuite
public void method1() {
System.out.println("before suite");
}

@BeforeMethod
public void method2() {
System.out.println("before method");
}

@Test
public void method3() {
System.out.println("test method 1 ");
}

@Test
public void method4() {
System.out.println("test method 2 ");
}

@AfterMethod
public void method5() {
System.out.println("after method");
}

@AfterSuite
public void afterSuite() {
System.out.println("after suite");
}
}

The class implementing the InvokedMethodListener interface:

import org.testng.IInvokedMethod;
import org.testng.IInvokedMethodListener;
import org.testng.ITestResult;
public class MyListener implements IInvokedMethodListener {

@Override
public void afterInvocation(IInvokedMethod method, ITestResult result) {
System.out.println("This method is invoked after every config method - " + method.getTestMethod().getMethodName());

}

@Override
public void beforeInvocation(IInvokedMethod method, ITestResult result) {
System.out.println("This method is invoked before every config method - " + method.getTestMethod().getMethodName());

}

}

TestNG.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IInvokedMethodListenerWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

• IMethodInterceptor: This listener helps to alter the methods that TestNG is supposed to run. It gets invoked just before TestNG invokes the methods. It just has one method name intercept that returns an altered list of methods.Let’s look at an example. The code here will run only methods with priority 1 in the test class. Other methods with different priority shall not be executed. This will be done after implementing the IMethodInterceptor listener.

  import org.testng.annotations.Test;
  
  public class IMethodInterceptorWithExample {
  
  @Test(priority=2)
  public void method1() {
  System.out.println("Method 1 will not be executed");
  }
  
  @Test(priority=2)
  public void method2() {
  System.out.println("Method 2 will not be executed");
  }
  
  @Test(priority=1)
  public void method3() {
  System.out.println("Method 3 will be executed");
  }
  
  @Test(priority=1)
  public void method4() {
  System.out.println("Method 4 will be executed");
  }
  }

  The IMethodInterceptor class implementing the interface:

  import java.util.ArrayList;
  import java.util.List;
  
  import org.testng.IMethodInstance;
  import org.testng.IMethodInterceptor;
  import org.testng.ITestContext;
  import org.testng.annotations.Test;
  
  public class MyListener implements IMethodInterceptor {
  
  @Override
  public List<IMethodInstance> intercept(List<IMethodInstance> methodsInstance, ITestContext testContext) {
  List<IMethodInstance> result = new ArrayList<IMethodInstance>();
  for (IMethodInstance method : methodsInstance) {
  Test testMethod = method.getMethod().getConstructorOrMethod().getMethod().getAnnotation(Test.class);
  if (testMethod.priority() == 1) {
  result.add(method);
  }
  }
  return result;
  }
  
  }

  TestNG.xml file:

  <?xml version="1.0" encoding="UTF-8"?>
  <!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
  <suite name="Parent_Suite">
  <listeners>
  <listener class-name="MyListener"/>
  </listeners>
  <test name="ItestReporter">
  <classes>
  <class name="IMethodInterceptorWithExample" />
  </classes>
  </test>
  </suite>
  <!-- Suite -->

  Console Output:

  

• IReporter: This listener helps to generate custom reports in TestNG, based on desired conditions. It contains a method called generateReport() which is invoked when all suites of TestNG are executed. The method uses three arguments:

1. 1. xmlSuite: It contains a list of suites for execution in the xml file
   2. suites: It contains all information about test execution and suites like class name, package name, method name and test execution results
   3. outputDirectory: It contains the path where the report shall be saved.

Let’s look at an example of how to customize reports through IReporter listeners. In this example, through the IReporter listener, the code shall run only methods belonging to a particular group. In this class, the group has been defined as ‘Sanity’, which shall be executed. The methods which are not part of the group shall not be executed. In the other class that implements IReporter, the generateReport() method has been used to customize the results accordingly. The customized results shall be visible in console with a corresponding report generated under the suite folder name specified in the testNG.xml file

import org.testng.Assert;
import org.testng.annotations.Test;

public class IReporterWithExample {

@Test(groups="smoke")
public void testcase1() {
System.out.println("This test case will pass");
}

@Test(groups="smoke")
public void testcase2() {
System.out.println("This test case will fail");
Assert.assertTrue(false);
}

@Test
public void testcase3() {
System.out.println("this tet case does not belong to the group smoke");
}

}

The class implementing the IReporter listener interface:

import java.util.Collection;
import java.util.List;
import java.util.Map;
import java.util.Set;
import org.testng.IReporter;
import org.testng.IResultMap;
import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.ISuiteResult;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestNGMethod;
import org.testng.ITestResult;
import org.testng.xml.XmlSuite;

public class MyListener implements IReporter {

@Override
public void generateReport(List<XmlSuite> xmlSuites, List<ISuite> suites, String outputDirectory) {
// TODO Auto-generated method stub
ISuite suite = suites.get(0);
Map<String, Collection<ITestNGMethod>> methodsByGroup = suite.getMethodsByGroups();
Map<String, ISuiteResult> tests = suite.getResults();
for (String key : tests.keySet()) {
System.out.println("Key: " + key + ", Value: " + tests.get(key));
}
Collection<ISuiteResult> suiteResults = tests.values();
ISuiteResult suiteResult = suiteResults.iterator().next();
ITestContext testContext = suiteResult.getTestContext();
Collection<ITestNGMethod> perfMethods = methodsByGroup.get("smoke");
IResultMap failedTests = testContext.getFailedTests();
for (ITestNGMethod perfMethod : perfMethods) {
Set<ITestResult> testResultSet = failedTests.getResults(perfMethod);
for (ITestResult testResult : testResultSet) {
System.out.println("Test " + testResult.getName() + " failed, error " + testResult.getThrowable());
}
}
IResultMap passedTests = testContext.getPassedTests();
for (ITestNGMethod perfMethod : perfMethods) {
Set<ITestResult> testResultSet = passedTests.getResults(perfMethod);
for (ITestResult testResult : testResultSet) {
System.out.println("Test " + testResult.getName() + " passed, time took " +
(testResult.getEndMillis() - testResult.getStartMillis()));
}
}

}

}

Below is the tesNG.xml for the classes to be run:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Report_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="ItestReporter">
<classes>
<class name="IReporterWithExample" />
</classes>
</test>
</suite>
<!-- Suite -->

Console Output:

A corresponding customized report shall also be created under the folder name which is the suite name defined in the xml file. In this example, it is Report_Suite.

Sample Report:

• ISuiteListener: As the name suggests, this listener works at the suite level. It listens and runs before the start and end of suite execution. It contains two methods:

1. onStart: invoked before test suite execution starts
2. onFinish: invoked after test suite execution finishes.

The following example incorporates the ISuiteListener. The code will use two child classes containing before and after suites. These shall be run through a TestNG xml file containing a reference to the ISuiteListener class which shall run its onStart and onFinish methods first and last respectively.

import org.testng.ISuite;
import org.testng.ISuiteListener;
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class MyListener implements ISuiteListener {

@Override
public void onFinish(ISuite suite1) {
System.out.println("onFinish function started of ISuiteListener " );

}

@Override
public void onStart(ISuite suite2) {
System.out.println("onStart function started of ISuiteListener " );

}

}

Below are the two respective child classes:

import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class ISuiteListenerWithExample {

@BeforeSuite
public void bsuite()
{
System.out.println("BeforeSuite method started for the first IsuiteListener example class");
}

@Test
public void test()
{
System.out.println("Test method started for the first IsuiteListener example class");
}

@AfterSuite
public void asuite()
{
System.out.println("AfterSuite method started for the first IsuiteListener example class");
}

}

And the second class:

import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import org.testng.annotations.Test;

public class ISuiteListenerExample2 {

@BeforeSuite
public void bsuite()
{
System.out.println("BeforeSuite method started for the first IsuiteListener example 2 class");
}

@Test
public void test()
{
System.out.println("Test method started for the first IsuiteListener example 2 class");
}

@AfterSuite
public void asuite()
{
System.out.println("AfterSuite method started for the first IsuiteListener example 2 class");
}

}

Their respective xml files:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="SuiteOne">
<test thread-count="5" name="Test">
<classes>
<class name="ISuiteListenerExample2"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite2">
<test thread-count="5" name="Test">
<classes>
<class name="ISuiteListenerWithExample"/>
</classes>
</test> <!-- Test -->
</suite> <!-- Suite -->

The final testNG xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Parent_Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<suite-files>
<suite-file path="ISuiteListenerWithExample.xml"> </suite-file>
<suite-file path="ISuiteListenerExample2.xml"> </suite-file>
</suite-files>
</suite>
<!-- Suite -->

Console Output:

• ITestListener: This is the most frequently used TestNG listener. ITestListener is an interface implemented in the class , and that class overrides the ITestListener defined methods. The ITestListener listens to the desired events and executes the methods accordingly. It contains the following methods:

1. onStart(): invoked after test class is instantiated and before execution of any testNG method.
2. onTestSuccess(): invoked on the success of a test
3. onTestFailure(): invoked on the failure of a test
4. onTestSkipped(): invoked when a test is skipped
5. onTestFailedButWithinSuccessPercentage(): invoked whenever a method fails but within the defined success percentage
6. onFinish(): invoked after all tests of a class are executedThe above-mentioned methods use the parameters ITestContext and ITestResult. The ITestContext is a class that contains information about the test run. The ITestResult is an interface that defines the result of the test.

Now, let’s look at an example showcasing the use of this listener.

Below is a listener class that implements ITestListener:

import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class MyListener implements ITestListener {

@Override
public void onFinish(ITestContext contextFinish) {
System.out.println("onFinish method finished");

}

@Override
public void onStart(ITestContext contextStart) {
System.out.println("onStart method started");
}

@Override
public void onTestFailedButWithinSuccessPercentage(ITestResult result) {
System.out.println("Method failed with certain success percentage"+ result.getName());

}

@Override
public void onTestFailure(ITestResult result) {
System.out.println("Method failed"+ result.getName());

}

@Override
public void onTestSkipped(ITestResult result) {
System.out.println("Method skipped"+ result.getName());

}

@Override
public void onTestStart(ITestResult result) {
System.out.println("Method started"+ result.getName());

}

@Override
public void onTestSuccess(ITestResult result) {
System.out.println("Method passed"+ result.getName());

}

}

The class below contains four methods, showcasing one method being passed, one method being failed, one method being skipped and one method being passed with a defined success percentage:

import org.testng.Assert;
import org.testng.SkipException;
import org.testng.annotations.Test;
import org.testng.asserts.SoftAssert;

public class ItestListenerWithExample {

int i=0;

@Test
public void testMethod1()
{
System.out.println("This method will pass and will invoke the onTestSuccess method of ITestlistener");
int i=10;
Assert.assertEquals(i, 10);
}

@Test
public void testMethod2()
{
System.out.println("This method will fail and will invoke the onTestFailure method of ITestlistener");
int i=10;
Assert.assertEquals(i, 11);
}

@Test
public void testMethod3()
{
System.out.println("This method will skip and will invoke the onTestSkipped method of ITestlistener");
throw new SkipException("Skipping this test case.");

}

@Test(successPercentage=50, invocationCount=5)
public void testMethod4()
{
i++;
System.out.println("Test Failed But Within Success Percentage Test Method, invocation count: " + i);
if (i == 1 || i == 2) {
System.out.println("this will be Failed");
Assert.assertEquals(i, 100);
}
}
}

Below is the testNG xml file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite">
<listeners>
<listener class-name="MyListener"/>
</listeners>
<test name="Listeners_program">
<classes>
<class name="ItestListenerWithExample"></class>
</classes>
</test>
</suite>
<!-- Suite -->

Console output:

Listeners play an important role when designing an end-to-end framework for web applications. They basically make execution smoother with required actions performed after events are triggered. From taking screenshots when test cases fail, to customizing reports to changing values during run time, TestNG Listeners help to refine and make automation testing easier.

Run manual and automated tests on real browsers and devices. Start running tests on 2000+ real browsers and devices on BrowserStack’s real device cloud. Run parallel tests on a Cloud Selenium Grid to get faster results without compromising on accuracy. Detect bugs before users do by testing software in real user conditions with BrowserStack. Get Started Free.