Test Management Revisited
The concept of test management sits awkwardly in agile, mostly because it’s a construct derived from the time when testing was a post-development phase, performed by independent testing teams. These teams were often of significant size, to reduce the time required to perform test execution. A test manager was often both line manager and the person responsible on projects for test strategy and reporting.
Agile, with its focus on cross functional teams, has sounded the death knell for many test managers. Today, you see the test lead role more frequently. This makes sense because the way we develop software has changed and it continues to evolve. Like any other discipline, testing must adapt to its new context.
Test Leadership
While test management is largely irrelevant in this world, there is still a desperate need for test leadership.
Why is this? The main reason is that as organisations struggle to become more innovative to respond quickly to market changes, engineering has responded by turning to continuous deployment and cross-functional teams to help meet demand. How testing fits into this picture is proving to be an Achilles heel for many organisations, which struggle to solve the challenge of how to making testing relevant and faster, yet uphold the quality they need to develop trust with their customer base.
The truth is, agile or not, most organisations adopt a testing approach constructed not long after the computer came into being—despite the enormous technological advances made in the last 70 years. We’ve developed innovative ideas on how to develop and release software, but when it comes to testing, the only real new idea since 1980 has been to automate test execution, or failing that, to outsource the problem.
It’s not that other ideas are not out there. Exploratory Testing and Context Driven Testing have been around a long time, yet while they are slowly gaining credibility and popularity in larger enterprise organisations (think Ebay, Barclays GTC), they are still a long way from becoming mainstream.
Organisations need experienced and seasoned testers who have the maturity, vision and courage to drag testing from the dark ages and pull it into the 21st century. Test Management is not going to do that, but Test Leadership might.
In the article “What Leaders Really Do” John P Kotter writes:
“Managers promote stability, leaders press for change.” He argues that managers focus on control, budget and planning, while leaders focus on aligning people, vision and motivation.John P Kotter
Testing needs leaders, not managers, to deal with these current challenges and help pull it into the 21st century.
We need test leaders working within organisations who have a vision of how testing can be better, and the courage to make those ideas a reality.
Banking, Disruption, XP
I’ve applied my ideas in test leadership when I worked first as QA Manager & then Head of Engineering at Tyro Payments one of Australia’s neobanks. I’ll explores the demands for testing in banking, how TP went from test management to test leadership and how that impacts the daily work of testers, how they take decisions, and how testers help teams to increase their test abilities, and how continuous delivery has impacted testing.
With good reason is banking a heavily regulated, risk adverse industry. Banks are entrusted with people’s money. Reliability, Security and Performance are key quality attributes and testing reflects this focus. Tyro Payments is a disruptor, a challenger to the traditional banking approach. We align ourselves in the fintech space, but we’re more tech than fin and our culture reflects that.
For example, our banking platform is developed in house using agile processes such as XP, pair programming and TDD. Everyone at Tyro Payments is responsible for quality and testing is an activity that all engineers perform. The tester on the team specialises in testing, using their investigative skills, their understanding of potential failure and how it might impact the customer. They experiment to better understand and learn the system. They share this knowledge with the rest of the team, helping the team make informed decisions about quality.
This ideology differs to traditional banking testing practice. In most banks, testing is seen as a control, the gatekeeper that ensures quality. At Tryo Payments, we’ve questioned this premise. For us, quality is a team responsibility. Instead we consciously work to build the best software we can given our professional capability and the goals & culture of the organisation we are working in. In this context, software testing is an investigation conducted to provide stakeholders with information about the quality of the product or service under test [Kaner 2006].
These concepts have changed how we view and conduct testing. For example, instead of a fixed upfront test strategy, we allow our test strategy to evolve and adapt as new risks such as bugs found, or new business directives, emerge. This allows us to quickly react to change, keeping our testing relevant to our stakeholders.
Another example is our focus on security. While we have a specialised security guild, Security is something everyone is responsible for. All engineers are trained to be aware of current threats and vulnerabilities.
This agile approach extends to how we work with regulators. Instead of second guessing what it meant to build a core banking platform, we worked closely with regulators, incrementally identifying through experimentation, the MVP required to obtain a banking license.
Test Leadership over Test Management
We’ve always challenged the premise that the test manager owns tasks such strategic direction, process improvement, hiring etc. Instead we distributed these tasks among the team. Testers had the option to chose areas that interested them. This resulted in them being responsible for recruiting, improving the testing process, internal training, social activities etc. This benefits the testing practice in two ways. It frees up my time, allowing me to focus on coaching and training, but also it provides opportunities for people to acquire new skills.
In the past year and a half, we’ve had rapid growth in team size, and made some significant changes to how we test. The team grew from 5 to 23 in just over one year. We embedded the testers into development teams, creating more cross functional teams. Recently, we’ve adopted the Spotify delivery model using tribes and squads.
With change and growth comes uncertainty, especially because we were trying out new approaches that many of us had not used before. We weren’t sure how to pair a developer with a tester. We thought it might be a good idea, and certainly we could see some benefits, but until we experimented and tested it out, we wouldn’t know.
Training and coaching was an imperative part to helping testers grow from tester to test leader, a skill needed if they were to successfully adjust and embed themselves into the teams.
We’ve explored many different ways to coach. We’ve worked out that for coaching to work it needs to be initiated by the tester. That means its optional to anyone who wants it. We’ve also worked out that a lot of the coaching is better performed with both tester and developer. Rarely does a tester work in isolation, so it makes sense for the tester and the developer to both understand the testing concepts.
The coaching is experimental in nature, typically focused on a task pertinent to both tester and developer. The style is Socratic to encourage participants to reason through the exercise, making the learning their own. There’s often a facilitated discussion after the task, helping to deepen any learning and also allow for explorations of ideas and concepts.
It’s this type of coaching that adds real benefit to how a tester is able to conduct themselves within a team.
Software Testing Strategy
It’s not what the testers do that matters so much as their mindset. Our testers don’t sit passively waiting to be told how and what to test. Instead, they provide input into the team from inception to release, making decisions along with the team on all aspects of testing. They are part of the team, and are treated as such. They give input on testing at planning meetings, while pairing with developers, at daily stand ups and retrospectives.
Each cross functional team is its own microcosm and consequently how testing is performed in each team differs slightly. The testers skill set varies too. More and more we’re seeing testers pair with developers, providing input into potential tests. But it’s not always the case. We’ve had testers focus on working closely with our business teams, helping them work through and create business process. We work to maintain close ties to the business so we can listen out for challenges they’re facing and feed that back into engineering teams.
What is consistent is that testers needs to be confident and self assured. They need to be able to know when to stand their ground but also when to let go. It’s this nuanced approach and understanding of team dynamics, along with great testing ability that in my opinion differentiates a test leader from a tester.
Agency in Software Testing
We hire testers giving them the autonomy to make any testing decisions they need. Most testers discover that involving the team in this decision making is a lot more powerful than making decisions in isolation.
Decision making can be a difficult process when the consequences of the decision have a big impact. Knowing what to consider, and the impact of the decision are two ways a coach can help a tester in this process.
One more thing. The result of allowing people to make decisions is that sometimes they will not make the ones you agree with. Any test leader will need to live with that. Decision making is a skill acquired through good and poor decision making. If you allow people to make decisions and then penalise them for poor decision making, the result you will get is disenfranchised testers or worse, testers unwilling to declare their interest in the decision making process.
Increasing Software Testing Skill
We’ve extended training on testing from a pure tester activity to one available to engineering. That means cross functional teams have training and coaching on testing available to them. Training on testing is extended to the whole team so that everyone can participate and discuss testing.
At inception, we have test engineers asking questions that prompt further discussion with the business. We help clarify and identify assumptions in stories making them easier to test down the path. Another way is to improve testability. Testability is how easy (or hard) software is to test. One of my favourite questions during planning meetings is ‘how will we test this’?
More and more it’s the team that’s working together to improve testing. For example, load testing credit card terminals can be a painstaking process. Often its the test engineer that ends up swiping a credit card again and again in the attempt to load the system. It’s a repetitive task, and pretty ineffective as a solution.
The test engineer brought the problem to her team. An innovative solution was developed where credit cards where hooked to a model train, allowing the process to be automated.
Continuous Delivery & Software Testing
We’re still in discovery mode with continuous delivery. We know that Continuous Delivery is going to have a huge impact on how we test but we’re still working out what this will look like. Fortunately, being a tech company we understand the importance of investing in the right infrastructure to support continuous delivery.
Being a bank, our risk profile means we want to fully investigate and understand the implications these new concepts have on banking. There’s not a lot of case studies on CD in a banking context! To fully understand the risk, we plan to work with both developers and operations, and to adopt an iterative approach, conducting low risk experiments to understand the impact.
We know we will need to test early and often and in production like test environments. We’ve already taken steps in that direction with teams owning production like test environments for testing prior to cutting a release. We working on building an immutable infrastructure, so how we conduct performance testing and exploratory testing in that context is undecided, but options include flags and feature switches to facilitate testing in production, parallel environments for performance testing, the use of synthetics and of course extensive monitoring in production.
Another area under consideration is the test data we use. Having ready access to controlled test data is going to play a big part in how effective our testing will be. This test data will need to be diverse in nature to hit lots of different business scenarios, but it also must be relevant to our customers. Understanding how our customers behave through production monitoring tools will play a large part in determining what data is required.
But this is only one point in time. In the future, the challenges that Tyro Payments and its team will face will change. Newer technologies such as AI become more prevalent in how we develop and test software. New ways to develop software will emerge, and we will shake our heads as we think of how we used to test in the bad old agile days.
Test Leadership has no easy algorithm or set of rules to follow that will ensure success. Being passionate about your vision helps of course, as does courage, when you face inevitable opposition. Empathy, can go a long way to understanding other’s perspectives.
Test Leadership is not for everyone but the reality is that for the majority of us working in technology, our biggest certainty is uncertainty.
Test Leadership may just help us prepare and perhaps even embrace this future.
Postnote: The quality strategy has evolved since this article (originally written for INFOQ) was written.
I’d like to thank, Andrei Charrett, Lauren Allen, Fiona Charles and Ben Linders for offering valuable feedback on this article.
Anne-Marie Charrett
Comments ()