How to optimize test cases for Continuous Integration
Shreya Bose, Technical Content Writer at BrowserStack - January 29, 2020
What is Continuous Integration (CI)?
Continuous Integration refers to a software integration practice that involves regular integration of code changes to a shared code repository.
Generally, integration occurs several times a day. This practice prioritizes committing small code changes frequently rather than infrequent big changes. Every commit triggers a build, which involves running a number of tests that identify if already existing features have been broken by new changes.
Advantages of Continuous Integration (CI):
There are numerous advantages that CI can offer to a software development process:
- Faster feedback: Since every code change triggers a series of tests, developers get almost immediate feedback on whether the latest change has caused a disruption in the existing code.
- Increased transparency: With tests being triggered automatically, the entire team would know if a bug pops up. This makes it easier to raise issues and identify anomalies.
- Easier integration: When code changes are being committed constantly, there is no point at which a major issue can go unnoticed. This means QAs have to work on minor bugs on an ongoing basis rather than deal with a cluster of code failures at one go.
How to optimize tests for Continuous Integration (CI)?
In order to successfully implement the practice of continuous integration, automated tests must be created and executed in particular ways. This is necessary for the tests to seamlessly fit into the CI pipeline.
- Create smaller tests
Smaller test suites run faster. Reorganize all necessary tests into smaller suites. This is especially necessary for tests that qualify the code for further testing. In other words, if these tests are not positive, then the system should not progress to further testing.
A good way to design these tests is: perform the most important operation first. For example, if a notepad-style app is being tested, then the first test should be automating the following functions:
– Create notes, add text, save, close app, reopen and verify that the right text has been saved.
If any of these functions fail, then there is a fundamental flaw that needs to be solved first. These build acceptance or build verification tests should be at the beginning of the test pipeline, They should be the first tests to run when new code is added to the codebase.
- Restructure the test setup
Tests begin with a setup phase, and then perform the steps for verification. In the notepad app example, the test checks if the app is saving a certain text after it has been reopened. To do this, the test has to be set up with the text first.
This setup phase can be restructured for better alignment with a CI-focused work ethic. For example, let’s say a team is running a suite of UI-driven tests that are time-consuming and lead to false failures because of timing and minor UI changes.
If the test setup is restructured to be performed via API commands and the verification is done through the UI, it will have the same functional coverage but will be executed significantly faster. The rate of false failures due to UI changes will also be reduced notably.
- Keep an eye on wait times
Imagine the scenario: a flaky test keeps failing because back-end elements did not respond in time. That, or a resource (possibly some external dependency) is still loading. Therefore, a sleep statement was put in as a temporary fix or workaround.
This works for a while, but if these statements are left in, they will cause unnecessary delays. Try replacing them with wait statements that complete when a predetermined event occurs, instead of ending after a fixed period of time.
- Initiate tests automatically
In a non-CI environment, test suites are normally initiated by human intervention. Modify these scripts so as to include them in the CI pipeline. This means setting them up to run automatically whenever a change occurs to the code repository.
Automate the tests that find problems which are usually difficult to fix. By identifying the problems sooner, the team will have more time to fix it. Such tests are very valuable but usually take some time to execute. Thus, automating them would also save the time it would take to initiate them manually.
- Use parallel testing
Parallel testing enables the running of multiple tests simultaneously on multiple devices and browsers. This is one of the most effective ways to speed up tests and create an effective CI pipeline.
Let’s imagine that a vital test suite contains five hundred test cases. This would take several hours to run in a linear fashion. However, if it is broken down into a dozen different suites that can run in parallel, the time taken is much lower. Consequently, the tests can be run more frequently, which results in a more thorough verification of code quality.
When a test suite takes too much time to execute, try breaking it down so that they can be run in parallel.
Optimizing tests for a CI pipeline is bound to significantly improve the testing process. However, remember that this requires the use of the right tools. Automated testing, in general, requires a robust test automation framework like Selenium that is, ideally, connected to a vast range of real browsers and devices.
This is where BrowserStack comes in. By providing a cloud Selenium grid of 2000+ real browsers and devices, it offers testers the opportunity to run tests as and when necessary with incurring the cost and effort of an in-house lab. BrowserStack also offers integrations with a vast number of CI/CD tools, thus making it the perfect platform to help optimize tests for Continuous Integration.