BlogRoll insight

50 Beta Testers. Not enough, or better than most?

Two software testers talked about the Joe Stump story this week.  One was Elizabeth Hendrickson and the other UTest.  The two attitudes to the same story are very different.
Let me compare two of the views:

First Elizabeth Hendrickson
“Problem #2: Thinking that “50 Beta Testers” and “200 Unit Tests” Constitutes Exhaustive Testing

Having beta testers and unit tests is a good and groovy thing. But it’s not sufficient, as this story shows. What appears to be missing is any kind of rigorous end-to-end testing.”


“So even the most cursory Exploratory Testing by someone with testing skill would have been likely to reveal the problem.”

Now UTest

“Not to lay blame at Stemp’s feet.  We, of all people, know that bugs happen.  Plus he mentions utilizing 50 beta testers and 200 unit tests, so they’re doing more testing than many.  But IF these showstopper bugs had been caught in the initial version or even in the 2nd version of Chess Wars, then Chess Wars wouldn’t have its nose pressed up against the window of the App Store, waiting for the powers-that-be to bless the new version.”

Whilst both blogs encourage more testing, I think I prefer Elizabeth’s attitude to the whole thing. It’s hard to know if 50 beta testers is sufficient or not without the context (last phrase included to avoid references to Unicorn Question in any know who you are…). Regardless, the attitude that  if you ‘do more testing than many’ somehow vindicates the fact that showstopper bugs were found is I think a misguided one.

As Elizabeth rightly pointed out in her post, good exploratory testing would have picked up this bug and any good tester knows to test for these things.

I hark back to my post on other approaches to testing for Startups. There are better ways to test! Lets not fall into the pitfall of accepting a lower level of testing as sufficient. Just because Startups use Beta Testing as the preferred modus operandi , doesn’t necessarily make it a good approach.

2 replies on “50 Beta Testers. Not enough, or better than most?”

50 Beta Testers can be both “not enough” and “better than most”.
In this particular case, we know that 50 Beta Testers, plus 200 Unit Tests was insufficient. The results tell us so.

Now we don’t know if more Beta Testers would be better.
And we don’t know if more Unit Tests would be better.

Elizabeth hypothesizes that testing by people with more testing skills would have been better (I agree).

uTest doesn’t really say. But of course they are in the business of selling crowdsourced testing, so it’s not hard to guess what they might suggest. (To me, their testing model is far more like “more Beta testers” than “more testing skills”. )

Hi Joe,
Elizabeth also hypothesizes that some ‘rigorous end to end testing’ be have been of benefit, something that I agree with too.

I think its hard enough to convince small companies & startups of the importance of some good pre-release testing without having ‘testers’ advocating this approach even if they are in the business of crowd-sourcing.

Leave a Reply

Your email address will not be published. Required fields are marked *