Advocacy for less testing

As a strategy, I'm a fan of minimising the number of end-to-end¹ tests a team owns and runs. However, this advocacy for less testing can be a difficult sell. Software Engineers and testers express their reservations about adopting such a strategy.

Good Reasons for End-to-End Tests

End-to-end tests are great. Here's why:

• They're relatively quick to create, especially if a separate team maintains the framework
• In a world where everything is mocked, it shows the happy path works.
• They offer value to non-engineering folks who can see something they know and understand working.  

Good Reasons for not having End-to-End Tests

On the face of it, there are lots to like about these tests. But I'm still a reluctant advocate. Here's why:

• Every test has a cost. Designing, building, running, evaluating, and maintaining costs time and expertise. Is that test worth it?
• How would you write your code differently if you didn't have these tests? Would you beef up your unit tests? Having isolated and decoupled tests may take more upfront thinking and care, but I believe it has a more significant payoff in allowing us to go faster.
• Be data-driven; how many bugs have these tests found post deploy? If your experience is that the count is low (when you remove false positives), then that's a super expensive safety net. Is it worth it?
• And if your end-to-end tests find bugs, are they catching bugs that should have been caught at the lower testing layer (such as unit testing or contract testing)?
• End-to-end tests only check what you know. It can't tell you about the unknown, unknowns. So in terms of a safety net, it's pretty poor.
• Even if the intent is to create a minimal set of tests, the lure of the "just in case" safety net is strong. Regression test suites get bigger and bigger as people are reluctant to 'delete' tests.

Deciding on a test automation strategy

Two sets of reasons and two perspectives. As a quality coach, how do I help move a team forward? The options are:

• Stand firm and exclude end-to-end tests from any testing strategy. After all, engineering has many standards that are absolutes. This is simply one more.
• Let the team decide. Stand back and allow the team to write as many end-to-end tests as they feel comfortable.
• Facilitate a discussion that allows everyone to explain their perspective, moving forward with an experiment.

I've attempted all three ways with various degrees of success. But, on reflection, I think all three options are good. What matters is that you are aware of yourself and the team's context, the constraints and resources a team has, the expertise of the team and the psychological safety.

These days I lean toward playing a long game where in time, teams discover a strategy that works for them. Because of this, I lean toward facilitated conversations and consensus-seeking.

Context Matters

One big reason for doing this is context. The team's context and more significant factors heavily influence software testing decisions. Look at these examples.

• A team using TDD and pair programming with vertical story slicing probably has greater confidence in the quality of their code. The need for a safety net is reduced.
• A team where psych safety is low, experimentation is seen as indulgent, and teams are blamed when a failure occurs is likelier to want strong and visible safety nets.

A word on psych safety

Blame and low psych safety will influence a team's decision without necessarily being articulated that way. Be mindful and empathic. Just as you hate waking up at 3 am with a pit of anxiety in your stomach, so does any team. Is it such a big deal if a few end-to-end tests give people a good night's sleep?

You matter too

But of course, your personal beliefs around testing matter too. And if it matters a lot, it's important to stick with that belief. And what about conflict? If you're about to trample over some sacred stones, does jumping into a mosh pit of conflict give your nerve endings a tingle of anticipation, or does it have you running for the hills?  

In summary, it's almost immaterial if you decide to go with or without end-to-end tests. Our tests don't exist in an isolated repo. They represent decisions on testing based on beliefs, relationships and safety. To change people's testing, you must change these three pillars. As a quality coach offering insight and support to their decision-making may be the most sensible strategy you can take.

¹   End-to-end tests in this context are GUI tests that hit the persistence layer.