A good automated test suite gives actionable feedback, helps fix bugs faster, and enables rapid delivery of software. A good automated test suite on the cloud does the same in a more cost-efficient way.
But more often than not, functional testing on the cloud ends up delaying feedback, increasing flakiness, and providing zero visibility into the failures—results which are completely at odds with initial expectations. It ends up hampering the pace of rapid delivery lifecycle.
To truly make the most of functional testing on the cloud, you have to reconsider the testing practices you follow.
At BrowserStack, we help thousands of customers run functional test suites on the cloud every month. Our QA and Support teams identified certain testing practices that are applicable to teams of all scales. Here they are, compiled into a 3-part guide which will walk you through the changes you’ll need to make in the setup, the test suites, and the solutions provider itself.
Part 1: The Setup
Architectures and system setups change depending on what you're optimizing for. 9 times out of 10, you'll be optimizing for speed, stability, costs, etc. These are not mutually exclusive.
A cost-efficient, more stable testing will also be one that lets you
- run tests faster (aim to finish the tests in under 10 minutes);
- re-run tests faster---whether it's a single test (for identifying flaky scripts) or entire suites after a bug fix;
- reduce flakiness;
- easily identify buggy commits, and
- scale with your tests, projects, and teams.
Here's how you can optimize your setup so it meets those requirements.
1.1 Target an acceptable test execution time
We want tests to finish within moments of starting, but that's outside the realm of possibility. A practical approach is to set a reasonable finish time for all your tests. Most people would target 10 minutes. This helps you plan better and identify unexpected failures or flakiness in the test setup.
1.2 Run your tests on every pull request (if not after every commit)
If your tests run within the acceptable time limit, you shouldn't have any problem running your tests frequently.
Cloud testing platforms today charge you based on the parallel threads you buy (i.e., the ability to run a number of tests all at once). This is why frequency won't affect the price you pay.
To isolate bugs early in the lifecycle, run tests after every commit. This can get overwhelming, so an alternative is to run tests on every pull request. The aim is to spread out testing and avoid a bottleneck towards the end of the release cycle.
1.3 Configure your CI for efficiency
You will want to reconsider your CI setup depending on your team's requirements, the frequency of testing, the projects you test in parallel, and the number of tests you intend to run. Most companies use a master-slave configuration in CI so that the master machine handles dispatching jobs, monitoring those jobs, and reporting the results. This is because a lot of jobs make the CI machine busy and unresponsive, leading to slow and flaky tests.
Lastly, ensure your machine can handle the number of requests that your tests make when they're run in parallel.
Network latency between your CI machine and your cloud testing platform can be minimized if your cloud vendor's data centers are close to you, geologically speaking.
Part 2: The Test Suites
Tests that run blazing fast on your workstation/device often take exponentially longer to run on the cloud. This is expected due to factors like network latency between your machines and testing platform, browser setup (or device startup) time, your webpage loading latencies, etc., There are a few things that you can do to speed up your tests considerably.
2.1 Keep the tests short
Keeping the tests short not only lead to fast execution times but also makes them less flaky (and easier to iterate or fix later). Best practices to keep in mind while writing the tests are:
- Skip testing elements that you don't need to test.
- Use Selenium requests sparingly. Each request increases your test execution time. (For instance, instead of an if-else condition to loop request until an elementID is received, try explicit wait).
- If the test case can be split into a bunch of smaller ones, split it.
2.2 Embrace parallelization
One excellent way to speed up your testing time is to run tests concurrently. Most modern testing frameworks support this right out-of-the-box. For instance, if you have a test script that tests 20 invalid inputs, split it into 20 smaller scripts and run them all at once.
2.3 Mix-and-match browsers to prevent queuing
Most cloud testing platforms set soft limits on their infrastructure to meet the demand at peak times, prevent abuse, and avoid denial of service attacks. Once you hit these limits, your tests will be queued to run afterthe current tests are done.
A simple workaround to queuing is "blending".
Blending is basically distributing your tests between the browsers---so more tests run in parallel without hitting the soft limit on any one browser. For example, suppose you have 100 tests that you must run on Chrome, Firefox, and IE each. Now, say your cloud vendor set a soft limit of 80 tests per browser type. At any time, you'll be able to run 80 tests on a browser and the remaining 20 will be queued. Instead, you can run the first 50 on Chrome and Firefox. Once this is done, run the remaining 50 on Chrome and Firefox.
This way, you minimize the time you'd have spent in the "queue".
2.4 Use built-in debugging options
Cloud vendors offer built-in debugging assistance---such as screenshots at every request (or every failure), video recording of the test session, Selenium logs, network logs etc. Instead of writing your own code and logic for taking screenshots and saving it locally as they take longer to run, use these features.
Tip: Where possible, organize your tests into builds and projects, and label them appropriately. Should a test suite fail, it would be easier to isolate the bug to a build and debug faster.
Part 3: The (Cloud) Solution Provider
Cloud gives you access to more browsers (types and versions) than your local workstation could handle. It gives you testing at an incredible scale. The trade-off for cost-efficient scaling is network latency.
To be confident of the code quality and functionality, there are ways to testing smartly on the cloud. Here are some suggestions.
3.1 Test on the right browsers and versions
Developers write code for and test on the latest version of Chrome. Now, this would suffice if all of your end-users are on the latest version of Chrome. But you'll have to do your cross browser testing across browsers
- that are favored by your end users,
- that have the highest usage trends in your target markets / countries,
- that are correlated with large drops in the conversion funnel (payments or signups, etc.),
These are browsers you need to prioritize your testing for, instead of every browser available on the cloud platform. Testing on additional browsers and versions won't delay your targeted test completion time—if you have enough parallel threads / concurrent slots with your cloud vendor. Otherwise, you might overshoot the targeted testing time considerably.
3.2 Test on different screen sizes
A product built for developers' use should not break on those huge monitors developers are fond of.
Your front-end developers spent days writing those perfect breakpoints. It'd be a waste NOT to test them on different screen sizes. If you expect your users to view your application on different screen sizes, be sure to test your responsive design.
3.3 Test on right mobile devices
Cloud platforms will have plenty of real mobile devices you can test on. At BrowserStack alone, we have over 2000 iOS and Android devices. That doesn't mean you should test on every last one of them---you'd just overshoot your targeted test completion time.
Just like with browsers and versions, find the mobile devices popular in your target market. Here's a quick blog post we published a while ago, on how to find the best mobile devices to test on.
3.4 Run tests in real user conditions
A lot of top-of-the-funnel conversions (like signups) take place on mobile. For eCommerce, over 40% of purchases are made through mobile devices. A buggy mobile app can—and will—directly affect your revenue.
Test a mobile app in real-world conditions. Configure timezones, IP addresses and locations, network speeds, etc. to mimic the environment your target audience is in. To find bugs that would affect real user experience, test on real devices instead of emulators or simulators, right before release.
3.5 Set up alerting, monitoring and bug tracking
Instead of waiting around till the tests are done running, you can set-up real-time alerts to get notified exactly when and where your test / build failed.
You can do this with a webhook from your CI machine or your cloud testing platform (or their REST APIs) to get the status of your test runs and set up notifications for yourself and your team. (We have a Slack integration for that, and more are in the works).
You can streamline your workflows further by integrating your cloud platform with your bug filing tool. These would automatically create tasks in your bug tracking tools---and populate them with all the data developers will need to recreate and fix. And if your setup tracks the bugs back to specific commits, you'll know who to assign it to.
Functional testing can be a pain. Functional testing on the cloud can be pain—with an added sting of network latency. This blog post lists some easy optimizations that'd not just help you test better, but also make the most of your cloud testing platform.
If you still haven't made up your mind on a cloud testing platform, you should give us a try.