Test Case Prioritization: A Detailed Guide

Quality of a Software can be assured by going through the Software Testing process of the Software Development Life Cycle (SDLC). Smarter Software Testing can help deliver a product, which is more reliable and defect-free meeting the business requirements and stakeholders’ expectations. This is the reason why it takes more time and resources and therefore, making this process very expensive. Due to the limited time left when the product reaches the Testing stage, it has become more important to prioritize the test cases, especially during Regression Testing in order to improve the efficiency of software testing.

Regression testing is a type of software testing, which checks that the changes, updates, or improvements made in the code base of an application does not impact the existing functionality of the software application. It is responsible for the overall stability and functionality of the existing features.

During this Regression testing of software, the Test Case Prioritization or TCP comes into play. TCP is one of the approaches to Regression testing apart from the Test Suite Minimization (TSM) and Test Case Selection (TCS).

Source

What is Test Case Prioritization (TCP)?

Test Case Prioritization or TCP, as the name suggests, is the process of prioritizing test cases in a test suite on the basis of different factors which could be anything ranging from code coverage, and functionality to risk/critical modules, features, etc.

It gives an approach to execute highly significant test cases first according to some measures and then produce the desired outcome such as, revealing the faults earlier and providing the testers with the feedback.

Why is Test Case Prioritization important?

• Testing is one of the most critically important phases of the SDLC which consumes significant resources in terms of cost, effort, and time.
• The difficult part of testing is the risk management, test planning, cost value, and being analytical about which test to run for your specific project.
• Running all the test cases in a test suite can require a large amount of effort and thereby, increase the regression testing cost.
• As the size of the software grows, the test suite also grows bigger and thus, requires more effort to maintain the test suite.
• For complex applications, it is impossible and impractical to exhaustively test each and every scenario.
• Prioritizing test cases based on perceived risks and customer needs can efficiently reduce the number of test cases required for testing an application.
• Prioritizing test cases also helps in meeting the project milestones along with meeting customer requirements and expectations.
• Early detection of bugs can also be achieved.

Priority Categories that can be used for Test Case selections

• Priority 1: The test cases, which MUST be executed, else the consequences may be worse after the product is released. These are critical test cases where the chances of getting a functionality disrupted due to a new feature is most likely possible.
• Priority 2: The test cases COULD be executed if there is enought time. These are not very critical cases, but can be executed as a best practice for a double check before launch.
• Priority 3: The test cases are NOT important to be tested prior to the current release. These can be tested later, shortly after the release of the current software version as a best practice. However, there is no direct dependency on it.
• Priority 4: The test cases are never important, as their impact is nearly negligible.

In the prioritization scheme, the main guideline that one must follow is to ensure that lower priority test cases should not cause any severe impact on the software. These prioritizations can have several goals. Some of them are below:

• Prioritization can be based on the functionality that is already communicated to the customers and is crucial from the business point of view.
• Gauge the likelihood of faults by checking out the rate of fault detection of a test suite. This helps understand whether a test suite is vulnerable or not.
• Increase the coverage of coverable codes of the system under test at a faster rate by allowing the code coverage criteria used earlier in the test processes.
• Increase the rate of detecting high-risk faults in a test suite, by locating similar faults earlier in the test process.
• Increase the likelihood of faults getting revealed due to specific code changes earlier in the regression testing process.

Types of Test Case Prioritization (TCP)

There are two types of test case prioritization:

• General Test Case Prioritization

In this prioritization, test cases are prioritized based on how they will be useful for subsequent modified versions of the product. It does not require any knowledge of the modified versions. Thus, a general test case prioritization can be performed following the release of a program version during off-peak hours. Thus, the cost of performing the prioritization is amortized over the subsequent releases.

• Version-specific Test Case Prioritization

In this prioritization, we prioritize the test cases such that they will be useful on a specific version of the product. This requires knowledge of all the changes that have been made in the product.This is performed prior to regression testing of the modified version.

What are different Techniques for Test Case Prioritization?

The five test case prioritization techniques used by QAs are as follows:

Coverage-based TCP

This type of test case prioritization is based on code coverage. Here the test cases are prioritized on the basis of their code coverage.

It has the following sub-techniques:

• Total Statement Coverage Prioritization: In this technique, the total number of statements covered by a test case is used as a factor to prioritize test cases. For example, a test case covering 5 statements will be given more priority over a test case covering 2 statements.
• Additional Statement Coverage Prioritization: This technique involves iteratively selecting a test case with maximum statement coverage and then selecting a test case that covers statements that were left uncovered by the previous test case. This process is repeated until all statements have been covered.
• Total Branch Coverage Prioritization: In this technique, total branch coverage is used as a factor for ordering test cases prioritization where branch coverage refers to the coverage of each possible outcome of a condition.
• Additional Branch Coverage Prioritization: This technique involves selecting a test case with maximum branch coverage and then iteratively selecting a test case that covers the branch outcomes, left uncovered by the previous test case.

Risk-based TCP

This type of prioritization uses risk analysis to identify potential problem areas which If fails, could lead to bad consequences. Therefore, the test cases are prioritized keeping in mind the potential problem areas. 

The following steps are performed in risk analysis :

• List potential problems
• Assigning the probability of occurrence for each problem
• Calculating the severity of impact for each problem

Read More: What is risk-based testing in agile?

Requirements-based TCP

In this technique, test cases are prioritized by evaluating them based on the different factors that determine the business requirements. These factors are documented in the functional requirement document and highlight the customer requirements in detail. Test cases are designed considering the customer assigned priority for a requirement, its complexity, and volatility while prioritizing. 

Each of the test cases is assigned a value based on these customer-centric factors, and then the test cases with a higher factor value are prioritized over those with lower values.

These are the factors used for test case prioritization:

1. Customer Assigned Priority (CP) – It is a measure of the significance of requirements from a customer’s business value point of view.
2. Requirement Volatility (RV) – It indicates how many times the requirement has changed.
3. Requirement Implementation Complexity (IC) – It indicates the effort or time required to implement a requirement.
4. Fault Proneness (FP) – It indicates how error-prone a requirement has been in previous versions of software.

History-based TCP

In this technique, test cases are prioritized primarily on the basis of the history of the test cases i.e. the past execution results are checked for the test cases. 

It is used to determine the possible chances of test failure and those which are more likely to fail are prioritized over those that have lesser chances of failure. The execution history of the test cases is used to select which test cases would be considered for testing in the current cycle.

Cost-Aware based TCP

In this technique, the test cases are prioritized on the basis of the cost factor. Test cases, which are less costly will get prioritized over the more cost-incurring ones. It includes:

• Cost of the regression testing process
• Cost of requirement gathering
• Cost of analyzing whether to select a test case
• Cost of prioritizing test cases
• Cost of entire test execution

Metric to measure Effectiveness of a Prioritized Test Suite

To measure the effectiveness of the prioritized test suite, you can use a metric called APFD (Average Percentage of Faults Detected) that can be calculated using the below formula

AFPD values can range from 0 to 100. Higher the APFD value, faster the fault detection rate.

So simply put, APFD indicates how quickly the test suite can identify faults or bugs in software. If the test suite can detect faults quickly, then it is considered to be more effective and reliable.

Conclusion

Undoubtedly, Test case prioritization is one of the most effective ways to produce high-quality products as per the market and end users’ standards. It is a way to prioritize and schedule test cases from high-level priority to low-level priority. This method is best to minimize the cost, effort, and time during the Software Testing phase. However, it is important to know the benefits, challenges, and techniques of prioritizing the test cases thoroughly to reap maximum benefits from it. This way you’ll end up prioritizing the test cases successfully.

No matter how you prioritize your tests, it is essential to test them under real user conditions to ensure better accuracy in the tests. By testing on real device cloud, like BrowserStack, you can increase your test coverage by getting access to 3000+ real browser-device combinations. This helps in ensuring faster and more accurate testing of the software. Run your Regression Tests using cloud Selenium Grid for parallel testing across different browser-device combinations for a seamless and consistent user experience.

Run Regression Tests on Real Devices