JUnit-XML or BDD-JSON based report upload

JUnit-XML reports have set a benchmark in test result reporting and are widely adopted across various testing frameworks. This standardization extends to several programming languages, including Java, JavaScript, Ruby, and Python, all capable of producing JUnit-XML formatted reports. Similarly, BDD-JSON reports are an important standard for Cucumber framework reports as they provide a detailed and easily interpretable format for test results.

You can create test reports using multiple frameworks through CLI and upload them to the Test Management from your terminal.

TestNG

Generate and import test results using TestNG

WebdriverIO

Generate and import test results using WebdriverIO

Nightwatch JS

Generate and import test results using Nightwatch JS

Appium

Generate and import test results using Appium

Cypress

Generate and import test results using Cypress

Mocha

Generate and import test results using Mocha

PyTest

Generate and import test results using PyTest

Playwright

Generate and import test results using Playwright

Espresso

Generate and import test results using Espresso

XCUITest

Generate and import test results using XCUITest

Cucumber

Generate and import test results using Cucumber

Understanding Test Run data handling

Test Run data selection using JUnit-XML reports

A JUnit-XML report typically includes details of automated test cases that were run as part of your test scripts. This includes Test Case titles, Test Case folder, Test Case execution time, Status (Pass, Fail or Skipped), and error responses in case of failures. The Test Runs that JUnit-XML reports represent aggregate individual Test Cases and their results. The report details the status, test steps, and results.

Data import mechanism

• Test Case Mapping:

  Test Case titles, Test Case folder, Test Case execution time, Status (Pass, Fail or Skipped), and error responses in case of failures for each of the Test Cases that are imported in Test Management.

• Creation or association:

  A Test Case is either mapped to an existing Test Case (by ID or Title) or a new Test Case is created.

JUnit-XML to Test Management mapping

JUnit-XML report tags is mapped to Test Management as follows.

• The test result status is defined by the child tags of the <testcase> tag. By default, the status is Passed. If there is any of the failure, error, or system-err tag, the test result will be Failed, and if there is a <skipped> tag, the test result will be Skipped.
• The time at the folder level is cumulative of each Test Case execution time.

Examples for a failed test, error response, and skipped test

Failed test example:

    <?xml version="1.0" encoding="UTF-8"?>
    <testsuite name="MyTestSuite" tests="2" failures="1" time="0.012">
    <testcase name="testMethod1" time="0.008">
        <failure message="Expected value was 10, but actual value was 5">java.lang.AssertionError</failure>
    </testcase>
    <testcase name="testMethod2" time="0.004"></testcase>
    </testsuite>

In this example, the test suite MyTestSuite contains two test cases. The first test case, testMethod1, fails because the expected value of 10 does not match the actual value of 5. The second test case, testMethod2, has passed. Therefore, in test management, MyTestSuite is mapped to a folder, testMethod1 test case status is set as Failed, and testMethod2 test case status is set as Passed.

Error response example:

    <?xml version="1.0" encoding="UTF-8"?>
    <testsuite name="MyTestSuite" tests="1" errors="1" time="0.005">
    <testcase name="testMethod" time="0.005">
        <error message="Error reading file: /tmp/myfile.txt">java.io.FileNotFoundException</error>
    </testcase>
    </testsuite>

In this example, the test suite MyTestSuite contains one test case, testMethod. The test case fails with an error because it could not read the file /tmp/myfile.txt. Therefore, in test management, MyTestSuite is mapped to a folder, and the testMethod test case’s error message is copied to the Response field of the test case.

Skipped test example:

    <?xml version="1.0" encoding="UTF-8"?>
    <testsuite name="MyTestSuite" tests="2" skipped="1" time="0.003">
    <testcase name="testMethod1" time="0.001"></testcase>
    <testcase name="testMethod2" time="0.002">
        <skipped message="Skipped test because it is not relevant to the current environment">java.lang.UnsupportedOperationException</skipped>
    </testcase>
    </testsuite>

In this example, the test suite MyTestSuite contains two test cases. The first test case, testMethod1, passes. The second test case, testMethod2, is skipped because it is not relevant to the current environment. Therefore, in test management, MyTestSuite is mapped to a folder, testMethod1 test case status is set as Passed, and testMethod2 test case status is set as Skipped.

CI/CD References

1. Create and import test reports using Jenkins.
2. Create and import test reports using Azure DevOps.