Common obstacles in Automated Testing
Shreya Bose, Technical Content Writer at BrowserStack - March 31, 2020
An effective test automation system is, by now, an absolute necessity in software development. In its ideal form, the system runs tests in parallel on multiple devices, thus speeding up results. It also frees up developers’ time to formulate new test cases, develop tools for deeper testing and engage more actively in code review. Instead of running the same test for the fifteenth time, they can write new code and develop new scripts.
However, given how central automation testing has become to the process, one must be conscious of its pitfalls. These pitfalls or bottlenecks will almost certainly show up in automation pipelines, and being aware of them helps teams address these issues.
This article discusses a few common bottlenecks in automated testing.
- Prioritization: As more and more tests are added to an automated testing framework, it becomes overloaded. Tests are delayed, and when this piles up, enough test cases fail that test results miss timelines and cause friction in release cycles.
There are two ways to handle this:
- Increase the number of parallel tests or prioritize which tests must be automated.
- If the former is not an option, then the latter brings up multiple questions.
- Which tests need to be run each day?
- Which can be run only once a week, or even less frequently?
- Can some tests be reshaped to run faster?
If there are too many test cases in the system, such prioritization would involve significant time and effort. Developers have to run manual tests, investigate bugs and fix them. On top of that, prioritization takes a serious toll on their bandwidth.
- Communication and Collaboration: Automation testing cannot function without collaboration.
It certainly requires more communication among all stakeholders than manual testing. In order to decide which tests need to be automated, a strategy must be discussed with developers, business analysts, project managers, technical architects and even a sample group of customers. Since automation tools are a significant investment, the plan, scope, and timeframe of the test strategy must be as specific as possible from the start.
Establishing lines of communication and collaboration among such a large number of individuals and departments is, needless to say, a challenge. Plus, to get all team members to agree on identifying automation objectives and setting targets, developers and testers need to spend time collecting evidence of effectiveness, historical data and sometimes even create a proof of concept.
- Finding the right tool: Automation testing cannot occur without the right tools. Given that there are a variety of tools in the market, choosing the right one poses a challenge. Tools vary from open-source options like Selenium to paid options. Each tool is suited to particular scenarios.
Depending on the nature and extent of automated testing, teams will have to do research into the tools, and also consider the available budget before making a choice. Simply choosing a popular commercial tool because others in the industry are doing so often backfires. An effective assessment involves defining tool requirements based on the application under test (AUT) as well as testimonials from experts who have already used the tools.
No matter the testing strategy and methodology, they will not succeed if the tool executing both do not match technical and business requirements. Without the right tool, any automation efforts are bound to fail.
A key part of this automation testing stack is the grid of devices used. Testing can never be comprehensive, unless the tests are run on real devices, to mimic real user conditions. Setting up a real device lab is expensive, cumbersome and requires constant updations. Picking a real device cloud like BrowserStack could be a great option.
BrowserStack, in fact, provides a cloud Selenium grid that of 2000+ real browsers and devices. The grid allows users to perform automated Selenium testing on multiple real browser-device-OS combinations, ensuring that websites and apps are optimized for usage in the real world.
- Finding the right skills: Just like automation cannot operate without the right tool, the tools cannot be operated without people with the right skills. It would be wrong to assume that manual testers or any technical testers can use automation tools if test scripts and recorded and played back.
Most test automation tools are only effective when manned by individuals who can accurately design and maintain automation frameworks, build solutions, test scripts. They must also be able to resolve emergent technical issues. Since automation frameworks can differ in terms of features and operability, they usually need to be versed in the framework’s design and implementation protocols.
Obviously, finding people with solid automation skills requires monetary investment. This may not be feasible for every organization, especially if they are looking to scale up significantly in a short span of time.
- Upfront investment: The initial phase of implementing automation is almost always expensive. One must analyze, design and choose or build an automation framework, libraries, reusable functions and the like. Certain cases may involve licensing costs on top of the usual operational costs. Even if one is using open source tools such as Selenium or Appium, one must take into account the investment going into training testers in using these frameworks plus the cost of setting up a device grid.
There are also hidden costs to consider. As explained before, setting up collaboration practices is integral to automation success. This may accrue costs in organizations that are just moving into automation practices. Having said this, automated testing always offers huge payoffs in the long-run and is the right choice for companies to scale up.
While it wouldn’t be right to say that the above are disadvantages of automation testing, they certainly are factors that must be considered before shifting to automated testing practices. The shift might require investment, education, and effort in its setup stages, but it has repeatedly proved to be worth all three. By establishing automation testing in software testing lifecycles, organizations will definitely be able to release better quality faster and with greater efficiency.