Optimizely Lowers Developer Pain by Moving Testing to the Cloud

Optimizely achieves testing maturity with BrowserStack’s scalable cloud infrastructure


Engineers frustrated with the on-premise grid 

Optimizely is the world’s leading progressive delivery and experimentation platform, enabling businesses to drive up the value of their digital products, commerce and campaigns. Product, engineering, data scientists, and growth marketing teams use Optimizely to quickly and safely rollout and rollback code to subsets of users through enterprise-grade feature flagging to ensure they can ship every new product, feature, or experience with confidence. This makes it critical for the platform to run smoothly for every business, and the engineering team ensures this by deploying high-quality releases at a high velocity.

The engineering team at the time was getting increasingly frustrated due to issues with their automated testing. Optimizely even has an in-house metric for this—Developer Pain Index (DPI). Brian Lucas, Senior Staff Software Engineer at Optimizely explains, “DPI is a mathematical function comprised of the time it takes to run the entire test suite over the average success ratio. The goal is to see it shrinking over time.”

Developer Pain Index

Testing at Optimizely was initially done on their on-premise grid. They started with PhantomJS-based headless browser tests and a limited number of browser-based tests. To avoid this, they began writing an end-to-end test suite for real browsers. The team quickly discovered that real end-to-end tests are more complex and unreliable compared to unit tests for headless browsers, and thus, more prone to failure and non-deterministic results. Creating this end-to-end test suite, therefore, required all available engineering bandwidth for several months.

At the same time, keeping the on-premise grid up and running was a challenge. There were browser-sandbox incompatibilities, a time-consuming discovery process for browser-version updates, and dependencies on package maintainers.

Moreover, the on-premise grid was inefficient in highly parallelized test environments, forcing engineers to run tests sequentially. “You can quickly reach a max capacity and run into physical resource exhaustion issues with on-premise machines,” explains Brian. These bottlenecks, coupled with the added overhead of building and maintaining the on-premise grid, frustrated the engineers.

To resolve these problems, Optimizely chose a competitor cloud-based testing service. “We worried that if we didn’t provide advanced tools to our engineers, we might be sacrificing some aspect of our quality process.” But the service provider’s unreliable infrastructure and frequent downtimes slowed the testing momentum Optimizely needed to achieve their goals.

They eventually moved to BrowserStack, looking for a more stable solution.


Scalable cloud infrastructure for streamlined QA

Moving to cloud infrastructure freed up the engineering team. “A dedicated service provider (like BrowserStack) can maintain the infrastructure better than us. Instead, we could focus on building and running tests to get better coverage,” says Brian.

As a result, the team could focus their attention on crafting a robust end-to-end test suite. They built a highly reliable framework titled Behavely on top of a Gherkin-based system called Behave. They now have multiple CI environments running tests, and a CI/CD pipeline with multiple environments. They leveraged parallelization and other BrowserStack features to deploy bug-free releases more frequently, improving the overall QA process.


Improved quality, no maintenance overheads, happier engineers

“The move to BrowserStack has been a force multiplier across all teams. It is a core and critical part of our build and release infrastructure, and every engineer ultimately depends on BrowserStack in one way or the other,” explains Brian.

BrowserStack’s elastic cloud of browsers allows Optimizely to add capacity when required, without any maintenance overheads or resource constraints on their CI infrastructure. As a result, they now build and test much quicker. Today, 15,000+ automated and parallelized test cases are run on Optimizely’s CI infrastructure (many of these on BrowserStack) every 45 minutes and released to a pre-production environment every 4 hours.

BrowserStack features—such as video and console logs—helped to simplify the bug-detection process, boosting the overall code quality. “Our engineers could perform forensic analysis or diagnosis on their tests in case something went wrong. They were empowered to deploy hotfixes to particular releases before they reached our customers. This way, our engineers could perform comprehensive final reviews to confirm the quality of our code.” Similarly, they could also monitor the total time and effort spent on running tests and use the insights to optimize their test suite, making it more reliable.

Efficiencies brought in by BrowserStack have lowered the Developer Pain Index by a significant margin, particularly where many browser tests would be untenable to run on an individual machine. With BrowserStack, Optimizely is swiftly moving towards the ideal state of software testing with a high-coverage test suite, accelerated release frequency, and happier engineers.

Watch the webinar to hear the story directly from Brian Lucas, Senior Staff Software Engineer at Optimizely.






Progressive delivery and experimentation


App Automate


15,000+ automated and parallelized test cases are run on BrowserStack every 45 minutes

"The move to BrowserStack has been a force multiplier across all teams. It is a core and critical part of our build and release infrastructure, and every engineer ultimately depends on BrowserStack in one way or the other."
Brian Lucas
Senior Staff Software Engineer, Optimizely