Test executions

Track how many automated or manual tests you run over time, and compare the trend across teams, modules, and platforms.

The test executions widget is a line chart of test execution counts over time. It counts every execution, not unique test cases, so a test case that ran ten times adds ten to the count. Test run type sets whether the widget counts automated or manual executions. Use it to check whether your test volume is growing, and to compare teams, modules, or platforms on one chart.

Summary of widget data and visualization

Each line on the chart is one segment. A segment is a named set of filters, such as one team’s projects or one platform. The legend names each segment and gives it a color. The widget gallery lists this widget as Tests Executions.

The X-axis follows the dashboard date range, so the range you pick sets how much time each point covers. A 7D range plots one point per day. A wider range, such as 3M, groups executions into weeks, so a single point covers a longer period.

Key metric information and visualization

The following table describes each element of the chart:

Add the widget

Follow these steps to add the widget to a dashboard:

1. Open a dashboard and select Tests Executions from the widget gallery. The widget dialog opens with a live preview.

   

2. Enter a Widget name. To add a note for readers, click Add a description and enter up to 200 characters.
3. Under Test run type, select Automated or Manual.
4. Turn on the Show chart summary toggle to show the total count for the chart.
5. Under Segments, enter a Segment name and add the filters for that line. Click Add segment for each extra line you want to compare.
6. Check the preview, then click Save.

The following table describes each option in the widget dialog:

Test run type

Test run type sets what the widget counts. The setting applies to every segment in the widget, and Automated is the default. The following table shows what each type covers:

Switching Test run type resets the widget to a single segment and clears every filter, because each type carries its own filter list. Set the type before you build your segments.

Planned executions for manual runs

Show planned executions adds a line for the planned execution timeline of every segment. The toggle applies to manual runs only, and it is off by default. The line is named <segment name> - Planned in the legend.

A planned execution is a test case in a manual test run that carries a Planned Execution Date. To see this line, you must set that date on the test cases in your test runs. The widget then plots how many test cases were planned for each period. When no test case carries a date, the toggle adds nothing to the chart.

To set the date, open a manual test run, select the test cases, and click Add execution details. In the dialog, pick the Planned Execution Date and click Save. The full procedure is in Add execution details.

Segments

Segments are how you compare teams, modules, or platforms inside a single widget. Each segment is one line on the chart, and each carries its own filters. Every segment needs a Segment name, because that name labels the line in the legend.

Use the following controls on a segment card:

• Click Add segment to add a line.
• Click the copy icon to duplicate a segment with its filters.
• Click the delete icon to remove a segment. The icon appears only when the widget has more than one segment.

Available filters

Filters apply per segment, so two lines on the same chart can cover different data. No filter applies until you add one. To add a filter, click Add filter, select a field, then pick the values you want. After you add the first filter, Clear filters appears next to Filters. Click it to remove every filter on that segment.

Some filters have a match operator next to the filter name. It sets how the values you select combine. The default, Matches any one value (OR), counts an execution when it matches any selected value. When a filter also offers Matches all values (AND), that option counts an execution only when it matches every selected value.

To reuse a filter set from another widget on the same dashboard, click Import from other widgets. The filter list depends on the Test run type you selected.

Filters for automated test runs

Apply the following filters when Test run type is set to Automated:

The list also carries a Custom fields group for the custom fields defined in your projects.

Filters for manual test runs

Apply the following filters when Test run type is set to Manual:

Drill down for more information

The chart tells you how many tests ran. The breakdown pane tells you which projects those executions belong to. Follow these steps:

1. On the dashboard, hover over a point on the chart.
2. Click View breakdown. The Tests Executions Drilldown pane opens.
3. Read the segment name, the execution count, and the period at the top of the pane.
4. Read the Project table. Each row is one project with its execution count for that period, sorted from the highest count down.
5. Click View tests next to a project. That project’s test list opens in a new tab, filtered to the same period.

Interpreting test data

A rising line can mean wider coverage. It can also mean the same tests running more often. Use the following patterns to tell the two apart:

Next steps

• Test execution distribution splits execution work by the person who owns or ran it.
• Automation coverage widget shows how much of your suite is automated.
• Trend of test cases separates growth in coverage from growth in volume.
• Test Executions widget in custom dashboards covers the same widget on a Test Reporting & Analytics dashboard.
• Test Management-specific dashboard widgets lists every widget you can add.