The State of Agility – Testing in Production

Share on FacebookShare on TwitterShare on LinkedInShare via email

The State of Agility is an 8-week blog series
from the Centare agile practice.

As organizations constantly thrive to offer superior quality product and extraordinary support for their customers, some of them even go to an extent of making sure their application can handle surprises by actively testing in production. For some organizations this may still be an unexplored or even a daunting concept. Regardless the development methodology, Testing in Production (TiP) is a powerful technique if explored and executed properly. TiP not only offers live feedback about the current state of the application in production, it also provides clues about some potential risks. Needless to say, TiP requires proper planning and solid technical and customer service stand-by support to handle outcomes.

Below are some of the many tests that can be performed in production.

Canary Testing

This term comes from the ancient coal mining days where the miners used a canary as the early warning for detecting carbon monoxide in the mines. When the canary stops singing, it’s time for the miners to get out of the mine.

In the software industry, a similar technique is to introduce a new feature to a very small subset of end-users (canaries) without their knowledge.  This takes some technology to accomplish, and is very useful when the change is risky and/or early testing on end users is needed to validate the new feature.  Since the code is released to a very small number of users, the impact of failure can be better managed.

To account for the possibility of failure, think about establishing a plan-of-action as it relates to the new changes.  For example, organizations often monitor the application performance, loads, hardware performance, service requests, and usability of the new features in real time during roll-out.  If the code satisfies the initial canary tests, then it can be gradually ramped up and released to other users.  Think about starting with a very small subset of users, less than 1%, and monitoring the application closely as you expose the new feature to more and more users.

Google performs canary testing on their Chrome browser releases.

Recovery Testing

Recovery testing is to test an application’s ability to recover from hardware failures, application crashes or other catastrophic problems.

It goes without saying that it is inevitable for organizations to think about a recovery process for all their applications in production. Most organizations start with documenting disaster recovery procedures, and maybe move on to actually testing it in a test environment.  These are great first steps, and eventually we want to know that production can actually handle disasters too.  That’s why organizations are proactively performing recovery testing by bringing down their live productions applications.

This article from Netflix explaining their TiP is a perfect example for proactive recovery testing and automating solutions to address the problems found during the testing.

A/B Split Testing

A/B split testing is used to test applications with similar content but slight variations, in order to discover which variation yields the best results.  We release these two variations to equally sized groups of randomly selected users.  Many times, this testing is performed to understand the user behaviors, priorities, content effectiveness, time they spend on a page etc.

Examples may be:

  • which groups have more new users sign-up
  • which group actually proceeds to the “read more..” page
  • which group  clicks “Buy now” button and actually proceeds to buying
  • which page title attracts more visitors

When done on a regular basis, cumulative results and observations yield valuable information over time that may dramatically dictate the web page look & feel and content.

Facebook and Google often use A/B testing. 


For ages, one of the common complaints about major testing challenges always included lack of production data and the hefty cost to mimic a production-like environment for testing purposes. Nowadays, as teams are moving to agile, they are expected to release to production with the highest quality. Nevertheless, I see the tests mentioned above are valuable to be performed in production for the following reasons:

  • to perform controlled release
  • to test disaster recovery
  • to gather market feedback

Any testing is performed to help the teams inspect the outcomes and adapt based on it. And I see TiP as one such way to inspect and adapt.

Pradeepa is a senior consultant in Centare’s Agile Practice. Her specialty is in quality assurance and agile testing using Scrum. Her passion is to take agile methodologies and engage teams to build high-quality software that meets business needs.

Tagged with:
Posted in Agile, Agile Management, Centare Blogs, Quality, Uncategorized
Job listings powered by the CATS Applicant Tracking System - ©2010 CATS Software, Inc.