In most companies, it is widely accepted that reducing risk before investing in new products or business ideas is a good idea. The risks we are trying to reduce are:
- value risk (whether customers will buy it or users will choose to use it)
- usability risk (whether users can figure out how to use it)
- feasibility risk (whether our engineers can build what we need with the time, skills and technology we have)
- business viability risk (whether this solution also works for the various aspects of our business)
See Marty Cagan here.
The three ways to do this are through research, analytics, and experimentation.
What seems less widely understood and less widely practised is the range of different experiments available to use and when to use them.
Beyond gathering insight, such as industry and competitive news, and analysing data the go-to “experiment” approach is talking to customers. While this is essential, how and when you do this need some thinking about.
Strategyzer (inventor of the business model canvas) has an incredibly useful experiment library, which they publish online for free. Their book Testing Business Ideas, digs into this in much more detail. They run a training course, which I did many years ago, which emphasised the need to go into the world and practice these skills – they pushed us onto the streets surrounding Liverpool Station in London to go and talk to people. However, applying these skills daily embeds them into people as a habit.
Customer interviews are what most people think of when it comes to validating an idea. They are good for testing desirability (demand), usability, and viability but are not used much in testing feasibility. Customer interviews are also fraught with risk. Teressa Torres’ book Continuous Discovery Habits – which emphasises the need to talk to customers – has much to say (backed by considerable academic research) about the dangers of doing so incorrectly. Customer interviews are cheap and easy to do. Still, the evidence is weak and potentially very expensively misleading if you ask the wrong questions and focus on asking what the customer wants regarding features instead of using them to understand customer problems.
Customer interviews are just one of 46 experiment methodologies Stratgyzer has identified. Knowing which one to choose in what context saves time and reduces risk. They distinguish between discovery experiments and validation experiments. In essence, discovery experiments help you understand the problem space and generate ideas, while validation experiments help you test and refine those ideas with more rigour.
Discovery Experiments:
- Purpose: To explore and generate new ideas or hypotheses about customer needs, problems, and potential solutions.
- Stage: Typically conducted early in the product development process.
- Focus: Broad and exploratory, aimed at learning and gathering insights.
- Methodology: Often qualitative, using methods like interviews, observations, and open-ended surveys.
- Sample size: Usually smaller, focusing on depth rather than statistical significance.
- Outcome: Generates hypotheses and insights for further testing.
Validation Experiments:
- Purpose: To test specific hypotheses or assumptions about the product, market fit, or business model.
- Stage: Conducted after initial discovery, often with a more developed product concept.
- Focus: Narrow and specific, aimed at confirming or refuting hypotheses.
- Methodology: More quantitative, using methods like A/B testing, controlled experiments, or larger surveys.
- Sample size: Generally larger to achieve statistical significance.
- Outcome: Provides evidence to support or reject specific hypotheses, informing go/no-go decisions.