How to test an idea ? Quick and cheap
The February 2009 edition of Harvard Business Review has an interesting article written by Thomas H. Davenport: How to Design Smart Business Experiments.
The objective of designing a simple test process for ideas is to avoid the two common pitfalls of acting on a hunch or overanalyzing, the middle ground being to quickly (and cheaply) reach a go / no-go decision based on significant evidence. Note that this simple test process is better suited for tactical decisions than for big, strategic decisions.
The author highlights the following steps:
- Create hypothesis: Examples of hypothesis would be "does project planning software improve our ability to meet deadlines for our projects?", or "does advertising in local industry publications improve sales of our web solutions for SME?" The hypothesis also contains an estimation of economic metrics, such as increased profits or savings that implementing the idea will achieve.
- Design test: Designing the test implies defining how to collect statistical evidence to test the hypothesis, such as how to execute the test, how to control execution, over which period of time, and what are the key metrics.
- Execute test: The test is carried on according to the design.
- Analyze test results: Test results are analyzed according to test design. One of the main outcomes of analyzing the results is to determine whether further testing is needed or not.
In my opinion, steps 1 to 4 should be performed iteratively. Few ideas can be tested in one pass. Therefore, analyzing test results can be used to refine the initial hypothesis. Of course, additional iterations are unnecessary for tests that resulted in a clear go / no-go outcome.
Finally, if the idea is deemed worthy, we have to plan and execute the rollout.