Testing in Production: A Detailed Guide

Let’s start with the obvious question.

What is Testing in Production?

Always be clear on this: testing in production is not releasing untested code in the hope that it works and/or waiting for the bugs to be detected once it is being used by end-users.

Testing is production basically lets a DevOps team prepare for possible bugs. It is common for bugs to end up in production. If the team is prepared to handle such bugs, then they can be resolved much quicker via continuous monitoring. This is what testing in production aims to accomplish.

Production testing can also be important for analyzing user experience. A/B testing is an important tool employed at this stage, which helps monitor how a new or recently updated feature is being received by the target users.

Why Test on Production when you can test on Staging?

• It is common for testers and organizations to consider staging environments to be mini replicas of prod environments. This can only be true if both environments are in sync. But, synching them brings up a host of challenges, such as:

1.  The fact that the staging cluster is usually much smaller than the production cluster means that configuration options for almost every service are going to be different. This applies to configurations of queues, databases, load balancers, etc. In case these configurations are stored in a database or a key value (Consul, Zookeeper), then auxiliary systems also need to be established in the staging environment to ensure that it interacts with these systems in the same way as the production environment.
2. Inadequate monitoring of the staging environment. Even if monitoring is enough, staging monitoring signals could be inaccurate since one has to monitor a completely different environment from the production environment.

• While it is crucial to test software at multiple stages of the development lifecycle, the conditions of the production environment are hard to fully replicate in staging (as explained above). This makes it impossible to provide complete test coverage for a website or app before deployment. This is especially true for edge cases that may remain untested in staging. They are most likely to display bugs when the code is deployed to production. Continuous monitoring of the production environment lets developers resolve any issues that may emerge from edge cases.
• Even if staging and production environments can be closely mimicked, certain tests yield the best results when performed in real web traffic. Take soak testing as an example. It verifies a service’s stability and reliability over a long time under realistic levels of load and concurrency, detects memory leaks, GC pause times, CPU utilization and more. The keyword here is “realistic levels” which cannot be generated in staging.

How to test in production?

Production testing can be conducted in two major ways:

1. A/B Testing: One way to test in production is to implement A/B testing. This basically means that two versions of a website or app or feature are released to gauge if users prefer one over the other.

For example, let’s say that some changes have been made to the “Cart” feature of an e-commerce website. To gauge if the changes lead to higher checkout rates, A/B testing is needed. It would involve releasing two versions of the site. Version A would be the original – with no changes to the Cart. Version B would be released with the changes.

Now the A/B test mechanism would record the checkout rates for both versions. If more users checkout on Version B of the site, then it is proved to provide a better user experience.

Since A/B testing can only be successful with real users, it is not possible to perform it outside the production environment. When done effectively, it provides much-needed feedback for developers/testers/other stakeholders.

2. Continuous Monitoring: By continuous monitoring of a prod environment after software deployment, teams can identify issues with the software that could only occur in prod. For example, identifying slow loading pages in a web application becomes completely different in prod. This is because a page may load easily in staging with a smaller data set and less traffic. In production, it is a whole other story.

Monitoring the time it takes for a web server to process requests will give real-world data about what users will face while using the site. Needless to say, slow page loading is quite disruptive to user experience and needs to be fixed as quickly as possible.

Best Practices for Testing in Production

• Use real browsers and devices. This might seem like an obvious statement, but it needs to be said. The production environment needs to be a real device-browser-OS combination. It is impossible to gauge how the software works without placing it in a real-world context. No emulator or simulator can replicate real user conditions, and therefore should not be considered as a viable option for testing.
• BrowserStack offers over 2000 real browsers and devices for both manual interactive testing and automated Selenium testing. They are hosted on the cloud which allows them to be accessible from anywhere at any time. One simply has to sign up, log in, and start testing for free.
• Time tests for when traffic is high. A truly successful application is one that runs flawlessly at the peak of usage. Since production testing intends to identify bugs in the real-world, it needs to occur in the most challenging conditions.
• Introduce a chaos monkey. According to Netflix engineer Cory Bennett and executive Ariel Tseitlin, “The best defense against major unexpected failures is to fail often. By frequently causing failures, we force our services to be built in a way that is more resilient.” A chaos monkey introduces failure into production randomly, thus making engineering build recovery mechanisms as well as stronger, more adaptive practices for bug resolutions. Refer to Netflix’s code on GitHub here.
• Monitor constantly and closely. When running production tests, keep a close eye on important metrics of user performance. If a major negative impact on user experience occurs, then be prepared to shut tests down quickly.
• If possible, give users the option to participate in experimental production testing. Ask users for their feedback on new features and releases by informing them clearly. By doing this, it is possible to collect end-user feedback, without major concerns about disrupting user feedback. When users know they are beta-testing a product, they won’t be as surprised or put off by bugs.

Testing in production is quickly becoming a non-negotiable part of the testing pipeline. With millions of users accessing a single piece of software from thousands of different devices, browsers, browser versions, and operating systems, it is impossible to predict and solve all bugs without real user experience. Production testing provides unique benefits to DevOps aligned developers and organizations. By allowing devs to be better prepared for dealing with anomalies, it contributes to creating better user experiences, brand credibility, and higher revenue. Undoubtedly, it is an essential aspect of software development in the current age.