Cost of Change in SaaS
Boehm's cost of change model has changed over the years. This article looks at the cost of change in SaaS and how it has changed the curve once again.
A history of the cost of change.
In the 1970s, Dr Barry Boehm hypothesised that the longer it takes to find a defect, the more expensive it is to fix it1.
Following on, the agile & XP community hypothesised that agile & XP practices would flatten this curve as smaller batch sizes and agile practices offered faster feedback loops.
There's been much debate about the accuracy and validity of the 'cost of change' model1. An interesting research paper on 171 software projects between 2006 and 2014 finds no credence to the hypothesis that finding defects early increases software engineering effort, suggesting that agile and new technologies have flattened the curve.
However, cost and effort are not always the same thing. Effort is only part of the picture. Other costs exist. For example, the test environments where defects are found also have a not-so-insignificant cost that we must also consider in the cost of change model.
Cost of Change in SaaS
In SaaS products, while delivering product quality is important, there's the knowledge and comfort that defects in production can be found and fixed far more rapidly than previously. With good monitoring and alerting, this additional safety net allows product discovery to continue innovating to determine customer value.
This impacts the cost of change model, as the cost of change in production is reduced.
The next sections explore the rise of recoverability, the impact of big data, and security threats on the cost of change models in SaaS products.
The rise of recoverability
As I said, good monitoring and alerting in production allow for faster recovery of any detected failure. Distributed tracing increases observability, aiding recoverability to the point where a customer is unaware that a defect ever existed.
Instead of considering rolling out a fix to multiple client environments, a SaaS company has only to consider their own production environment. This reduces the need to test and deploy on multiple operating systems and hardware environments, reducing the scope of testing and the complexity of releases.
Canary releases reduce the impact of failure, lowering the risk of failure and making it safer to deploy with less exhaustive testing.
The combined impact of SaaS, recoverability, observability, testing in production, and staggered releases reduces the cost of change in production and makes it safer to consider. This new safety net is why we can begin to consider reduced dependency on production-like test environments, which continue to rise in cost.
The Rise of Big Data
In SaaS, data is king, queen and knave. This is not just because companies rely on data analytics to improve customer experience and derive customer insights but also because metadata is a commodity on its own.
Owning and maintaining data is a commodity of its own. It's incredibly valuable in AI, with LLMs relying on datasets to train their models.
Securely storing current and historical production data with suitable redundancy is expensive. So much so that FinOps as a practice has emerged to help maximise the business value of cloud computing.
However, this is problematic for traditional software testing practices that advocate testing should be performed in test environments that mimic as closely as possible production environments. As the amount of data increases, so does the cost. Can a software testing strategy justify the cost of testing in these large environments when, as we've seen, it may be cheaper to rely on recoverability in production?
Increased security concerns
It's not only the cost of these environments that we need to consider. It's the threat test environments pose to a company's reputation if a test environment is breached and customer data is stolen.
Test environments are often targets for security attacks. This is because test environments typically don't have the same stringent policies as production environments.
To reduce the risk of security breaches, test data is often masked and obfuscated, preventing customer data from being viewed and increasing the cost of managing and maintaining the test environments.
A New Cost of Change Curve
In summary, the cost of change in production has reduced, while the cost of software testing in pre-prod test environments has increased.
And companies are starting to take note. In a recent company, one of the major strategies I was asked to drive was to reduce the dependency on systems integration test environments.
I believe this trend will continue to increase as more and more companies question the value of these heavy pre-production environments when, arguably, testing in production is becoming safer and is a better reflection of what the customer is experiencing.
Perhaps an aspirational SaaS model is one in which we attempt to make the cost of change linear. That is, we choose activities that offer the best value based on a risk cost analysis.
Reducing Dependency on pre-production environments
As quality professionals, we need to begin exploring approaches that enable us to perform good testing without being dependent on production-like environments.
Thinking critically through what quality is, the threats to quality and the strategies we can adopt to mitigate threats to quality will help us develop an approach that reduces dependency on these SIT (Systems Integration Testing) environments.
And there's much to look to for guidance. A good start is the prevent, detect and recover strategy that encourages us to think about approaches beyond testing and into prevention and recoverability can help us broaden and redistribute the risk of product failure.
SIT or NO SIT, you decide.
We don't perform software testing in isolation. In all of my work, I try hard to provide a balanced viewpoint that considers software testing and quality as part of product delivery. It's one of the reasons I work hard to understand the context and people's perspectives; in this article, the world is SaaS, and the people are senior engineering leaders such as VPs and CTOs.
But, like anything, we must be prepared to back ourselves and what we believe offers the most appropriate strategy for our product and our company. If you fundamentally believe that having a production-like environment is in the best interests of your company, be prepared to advocate for that. Cost alone is not a justification for retiring these environments. Make sure you provide evidence-based data that substantiates your reasons. Knowing what others may be thinking will help you develop a solid strategy that justifies your reasoning.
Good Luck!
1 Sacred Cows
It's probably blasphemy to say that regardless of its accuracy, the cost of change model is a valuable tool when discussing quality strategies with senior leadership.
But for blasphemy to exist, we must first believe in a sacred cow, and to be honest, software engineering isn't the type of profession that lends itself to sacred anything. We try things, and if they work, we continue to use them until they no longer work.
I find there's enough common sense in the cost of change model to make it useful as long as I treat it as a heuristic. I get that this isn't everyone's viewpoint, and some people would rather pluck their eyes out before mentioning its existence.
I'll continue to use it as it has served me well. If you wish to research the cost of the 'cost of change' model, here are some articles
Jim Bird
Comments ()