“You did great work on the scenario documents”, I said. They really were a lot better. The language was clearer, the instructions for the readers were better and the actual stories were more to the point. A major improvement, after having struggled with the scenarios for months.
“We decided to throw them out of the game altogether”, I continued.
Katie looked understandably confused. Didn’t I just say that they were much better? Had all the work gone to waste?
Making Playing Lean, the board game that teaches Lean Startup, has been a great challenge. Besides the actual playability of the game, we have had to make sure that there are enough startup lessons taught for the game to have value. Finding that balance had been a major part of the job.
That is why we introduced the scenarios. They were documents describing the beginnings of well known companies in a familiar industry. The first go was at Social Media. Players could be Friendster, Myspace, Facebook or Twitter. The outcome would be up to the choices players made, counterfactual history in the making.
Except something always felt wrong with them. It felt forced, we had to keep reminding players to read them out loud. Reading became a little awkward as the game progressed, but we could not really put our fingers on why.
We just assumed that it was because the quality of the documents themselves were so poor.
When the typos were weeded out, the structure was better and everything was much improved, it only took one playtesting to realise it: The scenarios had to die. The concept still felt forced and awkward.
Though the actual writing was discarded altogether, it lead to a major improvement in the game.
The broader lesson is that we fell into a trap. We convinced ourselves that the quality of our work wasn't good enough, that we could not rely on our testing until there had been major improvement. This made us discard a lot of of good feedback.
It may sometimes be the case that low quality experiments gives the wrong idea about an experiment. But be aware, it is a dangerous line of thinking. At the extreme end of it, only a fully functional, thoroughly designed and well engineered product can invalidate your idea.
It is important to be pay attention to what kind of people you are testing on as well. Some testers are more inclined to look past visual design and basic quality issues and evaluate the core idea of a product. Others won't get past the typos and poorly matched colors before they pass negative judgement.
Before my next experiment I will ask myself: What kind of people am I testing with? How high must the quality be before I will accept a negative result? It may just save me, Katie and the rest of the Playing Lean team a lot of time.
Twitter: @simenfurLinkedin: https://www.linkedin.com/in/simenfur
Website: www.playinglean.com
Bio: Simen is the CEO of Iterate, a consultancy based in Oslo, Norway. He is also the co-creator for Playing Lean, the board game that teaches Lean Startup.
