Automation Metrics

The Automation Metrics Dashboard provides insights into different CI Automation metrics like job frequency and historical trends. Get in-depth visibility into test-suite stability, and flakiness metrics for Testing-related jobs.

Build Stability

Definition

Build stability metric measures the stability of all test cases detected from your CI pipeline across different testing jobs. For an individual build run, this is calculated as the percentage of passing test executions from all test executions.

Metric Importance

• Efficient Development: Reliable test results enable developers to make informed decisions quickly, streamlining the development workflow.
• Confidence in Changes: Stable tests instill confidence in developers that their changes won’t break existing functionality, facilitating a more agile development cycle.

Potential Bottlenecks and How to Mitigate Them

• Flaky Tests: Identify and address flaky tests promptly. Analyze root causes and either fix or quarantine unreliable tests.
• Limited Test Coverage: Regularly assess and expand test coverage. Identify critical areas for automation and prioritize them in test development.
• Long Test Execution Times: Optimize tests for speed. Consider parallel test execution and focus on critical path testing to reduce overall execution time.

Jobs Stability

Definition

Jobs Stability metric measures the success rate of different jobs configured on your CI environment. This metric is calculated for jobs with different types and across different job runs for a given period.

Metric Importance

• Rapid Feedback: Stable CI jobs offer prompt feedback on code changes, helping developers catch and fix issues early in the development cycle.
• Efficient Development: Uninterrupted CI jobs allow developers to focus on coding without worrying about the integrity of the test and deployment processes.

Potential Bottlenecks and How to Mitigate Them

• Long Test and Deployment Times: Optimize test and deployment scripts for efficiency. Identify and address bottlenecks to reduce overall job execution times.
• Resource Constraints: Allocate sufficient resources to your CI environment. This includes CPU, memory, and network resources to handle concurrent jobs effectively.
• Inconsistent Environments: Inconsistent environments start affecting the overall Job stability metric. Use containerization tools like Docker to create consistent environments across different stages of the CI process.