quality quality engineering software testing

Quality and the Toyota Production Model

I saw this tweet the other day prompting this post on quality, visualisation and quality at pace. 

Understanding Quality

“Quality is value to some person” is a Jerry Weinberg quote emphasising the subjective nature of quality.   Tapping into an understanding of what your customers uphold, believe in and need goes a long way to creating a great product. It’s not easy to tap into what your customer’s value, but once you do, once you discover that “x-factor” you generate customer loyalty.   Michael Bolton added “Quality is value to some person who matters”. I and others have added “at some point in time” to indicate the transient nature of quality. This is the beauty and terror of quality. Its notoriously difficult to understand, let alone measure and quantify. Often we resort to quality attributes in an attempt to know and measure it.  How then does quality relate to the above tweet? What struck me was the underlined phrase: “Its about making problems….visible to workers and management, and addressing them as close to the source as possible” Breaking that down…

“Making problems visible….

Understanding and knowing what quality is for your team, your company and customers, makes it significantly easier to recognise problems that threaten its value. The difficulty is in knowing where these problems lie. If we knew where problems existed, their nature and size and their impact, making it visible would be much easier. Software testing helps us identify and shine a light on such problems. It gives us the knowledge we need to “know” if we are reaching the quality our stakeholders want. Poor old software testing though. Perceived as expensive and time consuming with every test having a cost. A cost in thinking about it, developing it, using it, evaluating the output, fixing any bugs related to running it, and retesting. But when performed as an investigation to discover threats to quality, software testing excels. It can discover problems no-one had dreamed about, one reason why many software testers are seen to have super bug finding powers. The reality is, we spend all are days thinking about how things might go wrong. And, like any skilled person it’s become a well honed craft.  By broadening the scope of our risk analysis beyond stories and product functionality, software testers can begin to exercise their powers of investigation and experimentation to many other areas. Many already help in story analysis to identify risk, but there are other places that potential problems lie. For example, performance, infrastructure, operations, test environment deployments, build times, bloated regressions test suites, flakey tests are all types of risks that potentially risk investigators can explore. I have often challenged and encouraged software testers on my teams to look beyond their bounded context and think of different types of risk. To create small experiments to investigate if a perceived risk is a real threat, and if it is, hand over to engineering to solve. The vital role of risk investigator starts to become real.  By rebranding a software tester to that of risk investigator, we can begin to see how this skill might be useful in the context of the Toyota Production Model where consistent visibility of problems is desired.

…to workers and management

It’s not enough for risk investigators to conduct experiments in isolation. That information has to be delivered to many different types of people whose concept of quality will probably differ and even conflict. Having data and being able to portray it in a way that is useful to that team is a real challenge. Conducting small experiments and providing as much technical data to other engineers is one thing. Facilitating senior management decision making is a different ball game (In the article Finishing Exploring I describe the subtle difference in providing data and supplying valuable information). I’ve always liked the idea of having a UX person create senior management infographic. I explored this concept at Tyro Payments when I was Head of Engineering there.

Visualizing Data by Anne-Marie Charrett Copyright 2017

For senior management, think about how you can portray information in a visible, colourful and easy to read manner. Use BLUF (Bottom line up front) and KIS (Keep it simple) principles avoiding dense text.

“..addressing them as close to the source as possible”

Early Feedback in software testing is not a “new” thing. The sound principle of unit testing has been around since I walked the hallowed aisles of Nortel Networks as a developer and a tester in the early nineteen nineties. What has changed is the technological advances that have enabled us to provide this feedback faster than before, and we want more of it. Long feedback loops introduced by end to end performance testing, or end to end GUI regression testing performed long after code has been written impact delivery. Unplanned work such as fixing bugs we didn’t know of disrupt this flow.   We’ve tried in the past to use automation in an attempt to speed up the feedback loop. That has worked to some degree, but its added additional work (read cost), and the value of some of these automated tests is debatable. Put it this way. If your end to end tests are not finding bugs that could be a problem. But if they are, that could be a problem too! By looking at alternatives to software testing that provide visual indicators of threats to quality throughout the life cycle we can begin to better understand the state of quality throughout delivery. I call this quality at pace.  More on that later

quality Training

Quality Workshop

One sure way to liven up a meeting is to ask attendees for a definition of quality. My experience has been that this is most entertaining particularly if you have both devops and testing in the room.
It turns out that many people feel very passionate about Quality. It also turns out that many have different ideas on what quality actually is. Who knew? Diversity of ideas are not bad but it can impact a teams understanding of quality. The consequence is that it could possibly hinder a team’s ability to delivery quality product.

To help people understand the challenges around quality and to try clarify what quality might be in their context, I developed the following quality workshop.

Purpose of Quality Workshop

a) to get a working definition of quality that most people are comfortable
b) to outline a manifestation of quality in terms of quality attributes
c) to identify ways in which we can these quality attributes can be acted on

Who should attend

Depends on what your trying to achieve, but if you’re a team delivering product, I would suggest your key stakeholders come along. That may mean product owner, developers, testers, Ops. If you can get your business there too that’s a big plus.

Tools for Quality Workshop

  • A set of Quality Attributes (I’ve known these also to be called Quality Characteristics, or CFR’s – Cross Functional Requirements) printed out and placed on a wall.
  • PostItNotes (of course!)
  • some pens.

Process for Quality Workshop

  1. Ask people to write down on a post-it note their understanding of quality
  2. Place the definitions on the wall and ask people to read them (or read them out)
  3. Note the diversity (if it exists, it typically does). Bring up the subjectivity of quality and how difficult it is to agree on and quantify.
  4. Ask what the implications of this might be throughout the delivery lifecycle (e.g Design, Development, Ops)
  5. You can dot vote to try and get some sort of agreement on a working definition of quality*.

* My working definition of quality is the Jerry Weinberg one “Quality is value to some person”.  If no-one puts this definition up on the board, I like to add it as it encourages a discussion on the subjectivity of quality. 

If you’ve got this far without some major outburst or spat, congratulations! Next step is to try and parse what quality might look like.

  1. Ask people to read the quality attributes on the wall
  2. Ask them to dot vote 3 what they consider to be the most important quality attributes
  3. Collect the top 3 quality attributes and put them on a whiteboard
  4. Ask the group to identify 3 tasks for each particular quality attribute and to write them on a post-it note.*
  5. Place the post it notes on the whiteboard along side the quality attribute
  6. Run through the post-it notes. Discuss the viability of these and how to make these tasks happen.
  7. Visualise and share this information to the broader organisation.

* I avoid using the word ‘testing’ as I want people to think about other factors that influence quality. Testing is important, but testing helps us evaluate quality.  Unless its acted upon, it doesn’t help us build quality. Sometimes I like to prime people by providing a couple of non testing examples such as Code Reviews,  Monitoring and Security by Design. 

I’ve found these sessions generate a lot of discussion and can become quite heated.

I wouldn’t run a quality workshop all the time, but its really useful iron out at perhaps an epic or feature level, what quality might be and how a team can work together to implement that vision.

(Learn more on this topic by attending my Quality Matters Workshop)

Let me know how you get on!

Photo Attribute:  Quality by jason Taellious

devops quality

Building the Quality Inn

Building Quality In integral part of moving to a DevOps culture, at least according to the State of DevOps 2016. But what exactly does that mean?
Sure, the report cites Deming, in particular

 “Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.”

And while the concept is not new, and seems to make sense at least at an ideological point of view, what in practice would it look like? This was what Matthew Santon Rutherford, the Quality Guild Doctor at IAG asked me.

He also mentioned that when people spoke of ‘Building Quality In’ it immediately conjured up this image of a run down motel somewhere remote, with a pink flamingo sign, swaying slightly in the wind.  We laughed and played around with this image for a while, adding an empty swimming pool and trying to imagine what each room might look like and who its customers might be.

I dedicate this series of “Building Quality INN” blog posts to Matthew, who has forever burned this image in my mind. What would your version of Quality Inn look like? Do you have a picture that might represent it? Please share !

insight jobs quality

My Testing Manifesto 2014

It’s my first day in permanent employment for, oh, about 20 years. I feel a little giddy, like someone’s first day of school – nervous but excited.
Unlike my first day of school, I have some clear goals and ideas I want to implement. I thought I might outline them here to help remind me of what I want to achieve and also to see how different it turns out.

Testing Philosophies

Testing is an skilled activity(not a phase) that all, to some degree can acquire.
Testers need autonomy to make decisions, to develop and perform excellent testing.
Quality is something we all care about, though it means different things to different people.
Every test has a cost (design, building, maintaining, reporting)

Goals for Testing

To develop a company wide reputation for excellent testing
To develop a test team that is able to handle testing problems with courage, skill and humility.
To coach and help develop the skill of testing with whoever may need it
To identify where testing is occurring and help augment that
To develop and the grow the test team in size, skill and expertise.
To engage with and support the testing community

Personal Goals

To acquire knowledge in testing within a continuous deployment, delivery environment
To learn more about functional programming
To be able to identify how to repair code
To work with integrity and within the bounds of what I consider ethical

agile quality

Why waterfall kicks ass

I read a blog post about why waterfall is NEVER the right approach and I feel compelled to respond to what’s touted as the waterfall mindset. Here’s a copy of the paragraph, but you can read the whole post on the above link to get a better sense of context.

I actually don’t believe adopting waterfall as an approach is ever a good choice.  Waterfall comes with the following mindset:

  • we don’t need feedback between the steps of requirements, analysis, design, code, test
  • we can hand work off
  • big batches are ok since they enable us to be more efficient
  • specialized skills working only on their specialty is good
  • we can understand the work to be done before we do it
  • written requirements can specify what we need

Putting aside for now, the use of absolutes, lets address this waterfall mindset:

1) we don’t need feedback between the steps of requirements, analysis, design, code, test

I’ve worked in both waterfall and agile over the years. In those ‘bad old days’ where no-one appreciated collaboration, we used to extensively review requirements. This meant that testers offered valuable input into requirements before any piece of code was developed. Since the invention of ‘agile’ almost everyone has discovered the 3 amigos, but honestly, this is not an agile concept, it existed way before agile was even thought of.

2) we can hand work off

Honestly, I don’t understand this sentence. I’m serious. I can think of many reasons why handing work to others is a good thing. For instance, if I’ve got too much work I’m in danger of becoming a bottleneck, and I hand some of my work over to someone else. If that’s a waterfall mindset, its one I like to have.

3) big batches are ok since they enable us to be more efficient

What do you mean by efficient here? Does it mean quicker, better quality, less waste? Efficient in what? Design, Writing code? Testing? Support? If I wish to carry 20 oranges from point A to point B, is it more efficient to carry them one at a time, or do I get a bag and carry them all together?Try delivering to a customer 1/4 of an IC circuit and request feedback? Sometimes delivering something in one batch IS the the more efficient way to proceed.

In waterfall, we did have teams, and work was allocated into smaller isolated tasks. It was rare one person developed the whole product. In fact, when I worked on telecommunication systems in the nineties, the concept of frameworks was being introduced, segregating data from its transportation method, allowing for people to work on separate parts of a system in parallel to each other.

4) specialized skills working only on their specialty is good

When I started working at Nortel Networks in 1994 it was all waterfall, except we didn’t call it waterfall then, we called it software development. Nortel Networks had a policy that encouraged developers and testers to spend time working in each others ‘specialisation’ area. For six months I became a developer working to deliver software. I was taught C++ and object oriented design principles, so I’m uncertain why you think this is a waterfall mindset?

5) we can understand the work to be done before we do it

Why is being able to understand work before you begin considered a ‘bad’ idea? I think this phrase is too ambiguous to be able to discuss with any merit.

6) written requirements can specify what we need

Is it the word requirement, or is it the fact that it’s written that makes this a ‘bad’ mindset? Yes we do have written requirements in waterfall. We also have written user stories in agile, so what is the point? Just because stories are written in confluence doesn’t make them any less written.

In the bad old waterfall days, I facilitated workshops with the business and IT teams to determine and understand risk. Lots of verbal collaboration, lots of whiteboard discussions.  I’ve worked in ‘agile’ teams were little communication takes place, no stand-ups nothing. In fact, one developer spent a week working on the wrong story without realising it.

I’ve worked on some fantastic waterfall projects that blew the socks of some really crappy agile teams I’ve worked with recently. In these waterfall projects, I’ve worked in harmony with great developers and testers, working closely together. We were afforded sufficient time to examine the system as a whole instead of its parts, something that these days appears to be a bit of a luxury. I’ve worked in environments that develop hardware, firmware and software where upfront design and ‘big batch’ systems thinking helped us understand that we were not merely developing code, but we were attempting to solve a problem.

Its easy to conflate poor software development practices with waterfall, just as its easy to conflate an agile approach with ‘good’ practice. To me the biggest change since those days are technological ones which allows us develop, integrate and compile it quickly and relatively cheaply. It’s the technology that has allowed us to develop in these small tasks that we are familiar with today. In the early nineties the concept of layering and isolating according to purpose was coming into play but we simply didn’t have the sophisticated systems that allowed us to develop in a way we wanted to.

A second important change was the conception of the agile manifesto. To me the agile manifesto is a stroke of genius. People who had the courage to espouse ideology that placed people over process, tools or artefacts. For many, this has changed how we think about developing software.

But lets not forget, that the agile manifesto was developed by people who worked in what you call a waterfall environment.  It seems to me these people had quite a different mindset than the one suggested above. It seems to me, those who developed the agile manifesto did so with ideas of collaboration and an emphasis on the ‘humanness’   of developing software. Do those ideas come about as a result of what was done badly in waterfall or because of what they saw was working well?

I suspect it was a little of both.

Don’t get me wrong. Lots of mistakes were made pre agile days. There was an idea of segregation between developer and tester in an attempt to avoid bias from developers. I’m glad we’ve gotten over that idea. But many of the mistakes we made in those days we’re still making now. Most software teams fail to understand testing and attempt to measure it using means that appear to have no direct correlation to quality. Many teams use measurement to lock down estimates as opposed to using them as a source of information for change. Go to an agile conference and count the number of talks on process and methodologies and frameworks. Look at today’s obsession with continuous delivery as a process. What happened to the people folks?

It’s easy to understand why. The agile manifesto is bloody hard to implement. Its much easier to point at a tool or a process and say “thats what we do” because its explicit, it’s easy to see. Programming & testing are human activities and are much harder to identify and talk about. It’s hard to describe and transfer these skills. The best way I know of is to actually perform the tasks.

We need a little more humility in acknowledging the great shoulders that agile stands on. Its simplistic to identify in hindsight a ‘waterfall’ mindset. Such a thing did not exist. Instead lets view them as the agile manifesto encourages us to do, to view them as people attempting to deliver quality software just like we are attempting to do today.

coaching quality test consultant Training

Please don’t hire me

If you want perfect softwareIf you want me to break your code
If you want to measure me by the number of test scripts I write
If you want me to tell you its ready to ship
If you want 100% of your product tested
If your looking for a tester to only “check” if things look ok
If you want your testers to mindlessly follow test scripts
If you want training on a test process
If you feel you can’t test without requirements

But if you’re looking for a tester who prides herself on the work she delivers, offers as much value to her clients as she can, and has a reputation for excellent testing, then yes, please hire me!

I’m available for coaching testers, training testers, consulting and testing.

Contact Details

insight quality

In search of perfection

I knew Flynn was in trouble the moment he created a program Clu 2 as his doppleganger. You see, the purpose of Clu 2 was to create the perfect system.

Flynn, a programmer failed to understand the struggle between perfection and quality. I’m guessing he didn’t spend a lot of time in the test team!

Of course, none of this makes any sense unless you have watched TRON the Legacy.

Warning!  I’m giving a bit of the plot away below…

In Tron Legacy, Flynn a programmer realises that he can’t spend all of his time in(yes in) his computer, so he creates Clu 2, a program that will create the perfect system for him. Unfortunately, the perfect system starts to wipe out everything that it sees as imperfect and pretty soon our world as we know it is being threatened. I know, its a pretty silly storyline, still the effects were great and it got the thumbs up from the 7 & 9 yr old critics.

What is perfection? Perfection, as I understand it, is  to be without fault or defect. A pretty tall ask for software. And Quality? Well, that is value to some person.¹

Both Quality and Perfection are subjective if you think about it. For example, art critics describe the Mona Lisa smile as the perfect smile. But in my mind, that small measly semi grin is far from perfect.

So, what is the difference between Quality and Perfection? Perhaps quality is more realistic, more humane?  They appear to be related in some way. When  some-one says something is perfect, are they perhaps saying that the quality is perfect?

Maybe perfection is a state in the quality model? A Utopian ideal that perhaps is something to aspire to as opposed to achieve?

At the end of the movie, Fynn realises that perfection(his son) was in front of him all the time (I told you the storyline was dodgy). I guess at that moment in time, blinded by emotion, his son was perfection to him. I suspect though, like any parent with inattentional blindness that moment quickly passes.

So perfection and quality  are dependent on time too. I think Markus Gärtner tweeted about that once.

How do we deal with these concepts in software testing? Here’s how I think about it:

Perfection is a great goal to aspire to, but my expectation is quality.²

I think this is a healthy way to look at it. For one thing, it stops me from asking for unrealistic demands from myself and others.

I do this by expecting good enough testing³.

I guess we all fall into this trap of perfection sometimes. its easy to demand perfection in other or systems yet excuse the imperfect in ourselves. In software testing, we expect perfection from developers yet don’t accept or recognise our own  failures.

“What do you mean its not a bug? Of course it is!”.

Expecting perfection in yourself is another trap and can set you up for some major life disappointments. A more realistic approach I think is to aspire for perfection but try to expect something a bit more realistic?Well, I try anyhow!

We need to combine this reality with a good dose of humility about our own failures and failures in others.

Then we will begin  treating  people with respect, a little more understanding, and perhaps then, our software will be more about the people, less about ourselves.



¹ Weinberg: “Quality is value to some person(s)”

² Read Secrets of a Buccaneer Scholar” for more on this.

³“Good Enough testing is the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product..