Black Box Testing

Like many young children, my niece Emily was a little scared of going into the ocean. It was a beautiful Irish June day, and my mother could see she wanted to join all the other children splashing about and having fun. She took her by the hand and led her into the water.

Shallow water can be very deep for some people.

There’s been some discussion about deep and shallow testing. While perhaps these distinctions provide value to people immersed in our craft, the concept of splitting testing to deep and shallow doesn’t make a lot of sense to our stakeholders, the people who matter.

Most of us are familiar with some form of  risk matrix matrix which categories risk into impact to the business and probability/likelihood of failure. We associate valuable testing with the red high range, yellow should test and low, nice to perform testing. For example, if the impact of the test is low, and the likelihood of a threat to quality being discovered is low, then ‘meh’ test if we have time.

Where does deep and shallow testing fit on this matrix? Well if you think of impact to a business user, then surface GUI testing is critical to them, so impact HIGH. The probability of failure? That too is contextual. It depends on the quality of the person coding, the clarity of what’s been asked for. But even if we are pretty confident that there is little probability of failure, it’s still in the ‘should test’ bracket due to the business importance. So business importance trumps probability.

Shallow testing is critical for business people. To call this testing shallow, negates the relationship we have to our customers and our business.

If we want to classify our testing, I suggest we do so in terms of how our business perceives it. System Testing, Functional Testing. Black Box and White Box Testing are ways of describing this testing without the inference of trivialness.

What do you think?

For initial thoughts on shallow and deep testing go to my blog post on

BOB series: Shallow Testing gets a bad rap

In my Exploratory Testing class, I talk about shallow and deep exploration. When you say the word exploratory testing, many think, ‘just playing around’ on the product. And to some extent that’s true, a good true. So when I talked about shallow exploration it seemed to

Anne-Marie Charrett (aka Maverick Tester)Anne-Marie Charrett

or try something different:

Quality is a journey - but do you know your destination?

After any decent exposure to software testing, Jerry Weinberg’s definition of quality tends to gain traction and respect. That is “Quality is value to some person”. You like it because of its subjectivity. You may also hate it because of its subjectivity. Remember that bug repor…

Anne-Marie Charrett (aka Maverick Tester)Anne-Marie Charrett