What is Test Observability?

Learn how you can debug faster with Test Observability to improve the quality of your automation suites

A sneak peak at Test Observability

What problems does Test Observability solve?

Test Observability attempts to find answers to the following questions faced by SDETs worldwide:

1. Among the tests that failed, what are the genuine failures and what is just noise?
2. Can I ignore all the noise and ensure that I don’t miss genuine failures without wasting time doing repetitive tasks everyday?
3. There are so many test failures but can I get to know how many unique errors is responsible for all of these failures?
4. Out of the failures that are not noise, can something help me carry out the root cause analysis in a matter of seconds?
5. I want to know the most unreliable or slowest tests in my suite.
6. I want to keep track of stability and performance metrics of my test suite over time and get alerted of regressions in behavior, and many more…

Why should you use it?

We’ve tried to list down a few reasons why we think Test Observability would be beneficial to you but the list is non-exhaustive:

1. Easy onboarding using SDK

Test Observability integrates with your existing test suite in minutes via the BrowserStack SDK, with zero-code changes necessary.

2. Debugging is made fun with smart detections

Smart detections of test flakiness , new test failures, always failing tests, test performance anomalies, and unique error analysis with automatic failure category detection makes real failure identification super intuitive and fast. Following are a few snapshots of the Build Insights from Test Observability that make life easy:

3. Integrated Testing Trends dashboard for all QA metrics

Test Observability calculates and consolidates all important QA metrics like stability, performance, flakiness, always failing percentages, frequency of runs, and many more into an easy-to-consume centralized Testing Trends dashboard for enhanced visibility.

4. Unique error analysis to detect common-cause failures

Test Observability analyzes all error occurrences in a build run, and groups similar failures together:

5. Intelligent & automatic failure analysis

Test Observability can detect test flakiness, new and always failing tests, test performance issues, and categorize failures for faster and more intuitive identification of real failures. The example below shows a test getting automatically categorised as a product bug:

6. Comprehensive test report with module level filters

Test Observability provides advanced reporting features that go beyond standard reporting. It allows automatic merging of reports from multiple CI runs or re-runs and provides a module level report view with a tree structure that displays historical test run status, tags, and open issues. The tool also offers rich filters and search options in a detailed test listing view and provides a quick preview of error and historical data for each test case.

7. Timeline debugging with consolidated logs

Test Observability provides a combination of Timeline and Time Travel debugging. With Timeline debugging, it consolidates logs from various sources such as framework, screenshots, browser’s JS console, network, and backend application, along with a video of execution, into a single timeline. This makes it easy to perform root-cause analysis. In addition, Test Observability allows users to travel back in time and debug at the source of an issue. An example of this is shown below:

8. Ability to re-run specific test cases on CI

Test Observability offers CI integrations that can be used to trigger re-run of specific test cases right from within the tool. The re-run results are merged back to the original run.

9. Jump to code or commit seamlessly

Test Observability integrates with SCM tools such as GitHub and GitLab, allowing users to access the source code or git commit related to a failed test case. The tool provides a deep integration with git, enabling mapping of the source code and test case execution. This allows users to easily access the test case code within the tool when needed.

10. Test Suite Health dashboard for all tests

Test Observability offers a test suite health dashboard that displays a detailed view of all tests that ran in a project, including their stability and performance regressions over time. The tool also provides x-ray views of test cases from a consolidated run standpoint, enabling users to analyze tests over a period of time to identify what needs to be fixed and determine the source of issues through a timeline view. An example view is shown below:

11. Top unique errors dashboard

Test Observability provides a unique feature that allows users to view all the unique errors in a project sorted by their occurrence rate or impacted tests count. This feature enables users to identify and prioritize fixing the most impactful errors to improve test suite stability. An example view is shown below:

Who is it for?

BrowserStack Test Observability is meant for anyone who runs and wants to analyze their automation tests. That said, these are the primary personas who will find the maximum use for the product:

• SDETs who run functional tests and wants to analyze and improve the quality of their suites.
• Developers who want to improve the quality of their unit testing suites
• QA managers who want to analyze the suite health issues to be able to know whats broken and need to be fixed.
• QA leaders who want to get a general idea about the testing trends in their QA org to find issues and plan ahead of time.

Supported frameworks

BrowserStack Test Observability is currently in private-alpha and it supports the following automation test frameworks but we’re building support for more frameworks at the speed of light:

• WebdriverIO
• MochaJS
• TestNG
• Nightwatch.JS
• JUnit XML Report Upload

What next?

Test Observability works for all your tests doesn’t matter where they run. Simply put, you need not run your tests on existing BrowserStack grid (Automate / App Automate). Test Observability also works for tests that you run locally on your laptop or on any CI tool or for that matter any other cloud solutions provider as well.

Get started with Test Observability in minutes for all your tests.