Code Coverage vs Test Coverage : A Detailed Guide
Shreya Bose, Technical Content Writer at BrowserStack - March 26, 2020
In software testing, code coverage and test coverage are important metrics. They are important benchmarks by which to measure the effectiveness of the codebase. However, these terms are sometimes used interchangeably, which they are not.
This article explains what each term means, and how they serve the developers’ purpose.
What is Code Coverage?
Code coverage is performed to verify the extent to which the code has been executed. Code coverage tools use static instrumentation in which statements monitoring code execution are inserted at necessary junctures in the code.
Now, adding instrumentation code does result in increased execution time and code length. But the increase is more than justified in light of the information that the tester gets because of the extra code.
Code coverage scripts generate a report that details how much of the application code has been executed. This is a white-box testing technique.
Why perform Code Coverage?
Code coverage is primarily performed at the unit testing level. Unit tests are created by developers, thus giving them the best vantage from which to decide what tests to include in unit testing. At this point, a number of questions arise:
- Are there enough tests in the unit test suite?
- Do more tests need to be added?
Code coverage answers these questions.
As development progresses, new features and fixes are added to the codebase. Obviously, the test code must be changed to stay updated with changes made to the software. However, testing standards established at the beginning of the project must also be maintained throughout subsequent release cycles. Code coverage can ensure that these standards are maintained, so that only the best quality code goes to production.
With a high percentage of code coverage comes a lower chance of unidentified bugs. In fact, one of the best practices in this regard is setting a minimum percentage of code coverage that must be achieved before going to production. This helps reduce the chances of bugs being detected at a later stage of development.
How To Perform Code Coverage?
Code coverage can be performed at different levels:
- Branch Coverage: This is used to ensure that every branch in a decision-making process is executed. Let’s say a tester is including a fall back for cross browser compatibility using an If…An conditional statement or a Do…While statement in the code. Branch coverage will ensure that all branches (If, Else, Do, While) are tested with appropriate input.
- Function Coverage: This ensures that all necessary functions are tested. It also includes testing functions with different input parameters to test the logic in the functions.
- Statement Coverage: In this, the code is created in a way that every executable statement in the source code is executed at least once. This includes corner cases or boundary cases.
- Loop Coverage: This ensures that every loop in the source code is executed at least once. Certain loops may be executed based on results achieved at runtime. One must be careful to test such loops so as to fortify the code completely.
- Condition Coverage: This reveals how variables in the conditional statements are evaluated. It helps to provide proper coverage to the control flow.
- Finite State Machine Coverage: This works based on the frequency of visits from static states and other transactions. Finite state machine coverage is the most complicated form of code coverage as it functions on the design of the software structure.
Code coverage verifies with instrumentation. Instrumentation monitors performance, inserts trace information, and detects errors in the source code. The types of instrumentation are discussed below.
Types of Instrumentation
- Code instrumentation: Source code is compiled after adding instrumentation statements. Compilation is best done using the normal toolchain. Successful compilation generates instrumented assembly. For example, to identify the time it takes to execute a particular function, add instrumentation statements at the Start and End of the function.
- Runtime instrumentation: These instrumentation statements collect information from the runtime environment i.e. when the code is being executed.
- Intermediate code instrumentation: Here, an instrumented class is generated by adding byte codes to the compiled class files.
Advantages of Code Coverage
- Quantitative: Code coverage offers results in quantitative metrics which helps developers gauge the nature and health of their code.
- Allows introduction of test cases: If already available test cases do not test the software extensively enough, one can introduce their own test cases to establish robust coverage.
- Easy elimination of dead code and errors: Let’s say some parts of the entire codebase were not touched during code coverage, or there are sections of dead or useless code. Code coverage allows easy removal of such code, thus improving the efficiency of the entire code base.
Now let’s explore test coverage.
What is Test Coverage?
Unlike code coverage, test coverage is a black-box testing technique. It monitors the number of tests that have been executed. Test cases are written to ensure maximum coverage of requirements outlined in multiple documents – FRS (Functional Requirements Specification), SRS (Software Requirements Specification), URS (User Requirement Specification), etc.
The test coverage report provides information about parts of the software where test coverage is being implemented. Essentially, it provides information about the tests executed on an application or website.
How To Perform Test Coverage?
Test coverage can also be evaluated through different types of testing. However, the type of tests that must be run depends on the business priorities of the testing team and the organization behind them. For example, user-centric web apps prioritize UI/UX tests over functional tests. Conversely, financial apps will prioritize usability and security testing over all other tests.
Some of the test coverage mechanisms:
- Unit Testing: Performed at a unit level or module level. Bugs encountered at this level are widely different from issues encountered at the integration stage.
- Functional Testing: Functions or features are tested against requirements mentioned in the Functional Requirement Specification (FRS) documents.
- Integration Testing: Also called system testing, since testing occurs on the system level. These tests are performed once all required modules are integrated.
- Acceptance Testing: Acceptance testing determines whether a product is fine to be released for customer use. At this stage, developers will have to receive a green light from the testers and SMEs, and then they can push code changes from the Staging to Production.
The purpose of test coverage often differs depending on the level at which tests are performed. It also depends on the type of product on which tests are run. Additionally, test coverage metrics for mobile phones would differ from those for website testing. Some classifications are:
- Features Coverage: Test cases are developed to implement maximum coverage of product features. For example, to test a phone dialer application the tester must ensure that the number being dialed is of proper length. If the number is Indian, it should have 10 digits. Otherwise, an error must occur. In similar fashion, all mandatory & optional features must be tested according to priorities set by the product team.
- Risk Coverage: Every product requirement document mentions the risks associated with the project, as well as how to mitigate them. They are addressed in this stage of test coverage. However, certain risks such as changes in market conditions, cannot be predicted or handled by this stage.
For example, while developing a business website, server infrastructure must be set up to ensure high-speed pages access. Depending on the location from where the website is accessed, the closest server must be chosen for loading the website. If not, the user gets low speed and their experience becomes sub-par. This needs to be tested. Similarly, the test team must perform load tests with a large number of users trying to access the site. The absence of such tests can result in adverse user experience.
- Requirements Coverage: Tests are defined to provide maximum coverage of the product requirements mentioned in various Requirement Specification documents.
For example, to test a pre-installed SMS application, the tester must ensure that the default language is set according to location. That means if the mobile is being used in a country where English is not widely used (such as Japan) the default SMS language should be Japanese.
Advantages of Test Coverage
- It reports on portions of the codebase that have not been covered by necessary test cases.
- It helps to detect the areas of test cases that are useless for the current project. These cases are reported and can be eliminated to make the code lighter.
- It helps developers create additional test cases as required. These additional test cases help ensure that test coverage is maximum.
Also Read: How to write Good Test Cases easily
Understanding code coverage and test coverage are necessary for software architects. These concepts help to clean, strengthen and refine code so that the resulting application is of the highest possible quality. By implementing these concepts, developers and QAs can create result-driven, sophisticated code that forms the basis of truly great software.