quality quality engineering software testing

Quality and the Toyota Production Model

I saw this tweet the other day prompting this post on quality, visualisation and quality at pace. 

Understanding Quality

“Quality is value to some person” is a Jerry Weinberg quote emphasising the subjective nature of quality.   Tapping into an understanding of what your customers uphold, believe in and need goes a long way to creating a great product. It’s not easy to tap into what your customer’s value, but once you do, once you discover that “x-factor” you generate customer loyalty.   Michael Bolton added “Quality is value to some person who matters”. I and others have added “at some point in time” to indicate the transient nature of quality. This is the beauty and terror of quality. Its notoriously difficult to understand, let alone measure and quantify. Often we resort to quality attributes in an attempt to know and measure it.  How then does quality relate to the above tweet? What struck me was the underlined phrase: “Its about making problems….visible to workers and management, and addressing them as close to the source as possible” Breaking that down…

“Making problems visible….

Understanding and knowing what quality is for your team, your company and customers, makes it significantly easier to recognise problems that threaten its value. The difficulty is in knowing where these problems lie. If we knew where problems existed, their nature and size and their impact, making it visible would be much easier. Software testing helps us identify and shine a light on such problems. It gives us the knowledge we need to “know” if we are reaching the quality our stakeholders want. Poor old software testing though. Perceived as expensive and time consuming with every test having a cost. A cost in thinking about it, developing it, using it, evaluating the output, fixing any bugs related to running it, and retesting. But when performed as an investigation to discover threats to quality, software testing excels. It can discover problems no-one had dreamed about, one reason why many software testers are seen to have super bug finding powers. The reality is, we spend all are days thinking about how things might go wrong. And, like any skilled person it’s become a well honed craft.  By broadening the scope of our risk analysis beyond stories and product functionality, software testers can begin to exercise their powers of investigation and experimentation to many other areas. Many already help in story analysis to identify risk, but there are other places that potential problems lie. For example, performance, infrastructure, operations, test environment deployments, build times, bloated regressions test suites, flakey tests are all types of risks that potentially risk investigators can explore. I have often challenged and encouraged software testers on my teams to look beyond their bounded context and think of different types of risk. To create small experiments to investigate if a perceived risk is a real threat, and if it is, hand over to engineering to solve. The vital role of risk investigator starts to become real.  By rebranding a software tester to that of risk investigator, we can begin to see how this skill might be useful in the context of the Toyota Production Model where consistent visibility of problems is desired.

…to workers and management

It’s not enough for risk investigators to conduct experiments in isolation. That information has to be delivered to many different types of people whose concept of quality will probably differ and even conflict. Having data and being able to portray it in a way that is useful to that team is a real challenge. Conducting small experiments and providing as much technical data to other engineers is one thing. Facilitating senior management decision making is a different ball game (In the article Finishing Exploring I describe the subtle difference in providing data and supplying valuable information). I’ve always liked the idea of having a UX person create senior management infographic. I explored this concept at Tyro Payments when I was Head of Engineering there.

Visualizing Data by Anne-Marie Charrett Copyright 2017

For senior management, think about how you can portray information in a visible, colourful and easy to read manner. Use BLUF (Bottom line up front) and KIS (Keep it simple) principles avoiding dense text.

“..addressing them as close to the source as possible”

Early Feedback in software testing is not a “new” thing. The sound principle of unit testing has been around since I walked the hallowed aisles of Nortel Networks as a developer and a tester in the early nineteen nineties. What has changed is the technological advances that have enabled us to provide this feedback faster than before, and we want more of it. Long feedback loops introduced by end to end performance testing, or end to end GUI regression testing performed long after code has been written impact delivery. Unplanned work such as fixing bugs we didn’t know of disrupt this flow.   We’ve tried in the past to use automation in an attempt to speed up the feedback loop. That has worked to some degree, but its added additional work (read cost), and the value of some of these automated tests is debatable. Put it this way. If your end to end tests are not finding bugs that could be a problem. But if they are, that could be a problem too! By looking at alternatives to software testing that provide visual indicators of threats to quality throughout the life cycle we can begin to better understand the state of quality throughout delivery. I call this quality at pace.  More on that later

quality Training

Quality Workshop

One sure way to liven up a meeting is to ask attendees for a definition of quality. My experience has been that this is most entertaining particularly if you have both devops and testing in the room.
It turns out that many people feel very passionate about Quality. It also turns out that many have different ideas on what quality actually is. Who knew? Diversity of ideas are not bad but it can impact a teams understanding of quality. The consequence is that it could possibly hinder a team’s ability to delivery quality product.

To help people understand the challenges around quality and to try clarify what quality might be in their context, I developed the following quality workshop.

Purpose of Quality Workshop

a) to get a working definition of quality that most people are comfortable
b) to outline a manifestation of quality in terms of quality attributes
c) to identify ways in which we can these quality attributes can be acted on

Who should attend

Depends on what your trying to achieve, but if you’re a team delivering product, I would suggest your key stakeholders come along. That may mean product owner, developers, testers, Ops. If you can get your business there too that’s a big plus.

Tools for Quality Workshop

  • A set of Quality Attributes (I’ve known these also to be called Quality Characteristics, or CFR’s – Cross Functional Requirements) printed out and placed on a wall.
  • PostItNotes (of course!)
  • some pens.

Process for Quality Workshop

  1. Ask people to write down on a post-it note their understanding of quality
  2. Place the definitions on the wall and ask people to read them (or read them out)
  3. Note the diversity (if it exists, it typically does). Bring up the subjectivity of quality and how difficult it is to agree on and quantify.
  4. Ask what the implications of this might be throughout the delivery lifecycle (e.g Design, Development, Ops)
  5. You can dot vote to try and get some sort of agreement on a working definition of quality*.

* My working definition of quality is the Jerry Weinberg one “Quality is value to some person”.  If no-one puts this definition up on the board, I like to add it as it encourages a discussion on the subjectivity of quality. 

If you’ve got this far without some major outburst or spat, congratulations! Next step is to try and parse what quality might look like.

  1. Ask people to read the quality attributes on the wall
  2. Ask them to dot vote 3 what they consider to be the most important quality attributes
  3. Collect the top 3 quality attributes and put them on a whiteboard
  4. Ask the group to identify 3 tasks for each particular quality attribute and to write them on a post-it note.*
  5. Place the post it notes on the whiteboard along side the quality attribute
  6. Run through the post-it notes. Discuss the viability of these and how to make these tasks happen.
  7. Visualise and share this information to the broader organisation.

* I avoid using the word ‘testing’ as I want people to think about other factors that influence quality. Testing is important, but testing helps us evaluate quality.  Unless its acted upon, it doesn’t help us build quality. Sometimes I like to prime people by providing a couple of non testing examples such as Code Reviews,  Monitoring and Security by Design. 

I’ve found these sessions generate a lot of discussion and can become quite heated.

I wouldn’t run a quality workshop all the time, but its really useful iron out at perhaps an epic or feature level, what quality might be and how a team can work together to implement that vision.

(Learn more on this topic by attending my Quality Matters Workshop)

Let me know how you get on!

Photo Attribute:  Quality by jason Taellious

insight quality

In search of perfection

I knew Flynn was in trouble the moment he created a program Clu 2 as his doppleganger. You see, the purpose of Clu 2 was to create the perfect system.

Flynn, a programmer failed to understand the struggle between perfection and quality. I’m guessing he didn’t spend a lot of time in the test team!

Of course, none of this makes any sense unless you have watched TRON the Legacy.

Warning!  I’m giving a bit of the plot away below…

In Tron Legacy, Flynn a programmer realises that he can’t spend all of his time in(yes in) his computer, so he creates Clu 2, a program that will create the perfect system for him. Unfortunately, the perfect system starts to wipe out everything that it sees as imperfect and pretty soon our world as we know it is being threatened. I know, its a pretty silly storyline, still the effects were great and it got the thumbs up from the 7 & 9 yr old critics.

What is perfection? Perfection, as I understand it, is  to be without fault or defect. A pretty tall ask for software. And Quality? Well, that is value to some person.¹

Both Quality and Perfection are subjective if you think about it. For example, art critics describe the Mona Lisa smile as the perfect smile. But in my mind, that small measly semi grin is far from perfect.

So, what is the difference between Quality and Perfection? Perhaps quality is more realistic, more humane?  They appear to be related in some way. When  some-one says something is perfect, are they perhaps saying that the quality is perfect?

Maybe perfection is a state in the quality model? A Utopian ideal that perhaps is something to aspire to as opposed to achieve?

At the end of the movie, Fynn realises that perfection(his son) was in front of him all the time (I told you the storyline was dodgy). I guess at that moment in time, blinded by emotion, his son was perfection to him. I suspect though, like any parent with inattentional blindness that moment quickly passes.

So perfection and quality  are dependent on time too. I think Markus Gärtner tweeted about that once.

How do we deal with these concepts in software testing? Here’s how I think about it:

Perfection is a great goal to aspire to, but my expectation is quality.²

I think this is a healthy way to look at it. For one thing, it stops me from asking for unrealistic demands from myself and others.

I do this by expecting good enough testing³.

I guess we all fall into this trap of perfection sometimes. its easy to demand perfection in other or systems yet excuse the imperfect in ourselves. In software testing, we expect perfection from developers yet don’t accept or recognise our own  failures.

“What do you mean its not a bug? Of course it is!”.

Expecting perfection in yourself is another trap and can set you up for some major life disappointments. A more realistic approach I think is to aspire for perfection but try to expect something a bit more realistic?Well, I try anyhow!

We need to combine this reality with a good dose of humility about our own failures and failures in others.

Then we will begin  treating  people with respect, a little more understanding, and perhaps then, our software will be more about the people, less about ourselves.



¹ Weinberg: “Quality is value to some person(s)”

² Read Secrets of a Buccaneer Scholar” for more on this.

³“Good Enough testing is the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product..