Automatic failure analysis

Learn how automatic failure analysis works and how it helps speed up debugging and RCAs

Pre-requisites for it to work

Automatic failure analysis is triggered after completion of each build run. But it will work only if:

1. There are failures in the current build run.
2. There should have been test failures in the same project, in the past.
3. You should have manually tagged test failures into one of the failure categories in the past.

How to tag failures into categories manually?

Initially, all failed tests would be tagged as To be investigated and you could click on the dropdown against the test name and select one of the following categories as shown below:

• Product bug
• Automation bug
• Environment issue
• No defect
  
  

How does it work?

Automatic error analysis for failure category detection works by employing a host of sophisticated algorithms. It analyses errors seen in the past and takes into account the category that you might have assigned manually to the different errors while doing your analysis in the past.

Automatic error analysis learns with every instance where you tag errors manually and over time it becomes more and more adept at intelligently categorizing issues. To add to that, you can also create your own sub-categories through Settings and assign different failures to those.

How does it help?

As an automation engineer responsible for the quality of features that go into production, the single most important thing is the to ensure that no real product bugs are able to creep into production and impact the end user’s experience.

In the process to ensure that above, automation engineers across the globe write tons of tests to verify that scenarios work fine. But, the biggest problem faced is that of flakiness and other non-product-code related issues which result in test failures.

Hence, it becomes super important for you to know which test failures are genuine vs the ones that are just noise. We have built the Automatic Failure Analysis feature keeping exactly that in mind.

Automatic Failure Analysis helps by doing the following:

1. Categorizes issues into Product bugs, Automation Issues, Environment issues for you to identify which ones you can ignore from analysis.
2. You can pin point test failures due to environment issues and it helps you with the next course of action for e.g. re-run those tests.
3. You can also get a list of all Automation issues and either Mute them or create an issue on a Project Management tool such as Jira.

Configure parameters of automatic error analysis

You can configure settings for automatic error analysis and control how it works and whether it works.