Many CRO experts are hardcore believers in statistical significance. And I am too.
It’s fantastic to make data-driven business decisions based on real world evidence. Marketers guess too often.
But where the hardcore reliance on significance can let you down is when it becomes the driving factor in decision making. I see people holding out for a specific level of significance all the time and it distracts from the core mission – incremental growth.
The truth is, there’s nothing magical about 95% statistical significance, 99% or even 75%.
So when it comes time to deciding whether to end a test or keep it running to reach a magical significance level, it’s important to assess risk.
As CEO of New Republique, Nima Yassini writes, there is risk involved at every level of significance.
There’s also a massive cost in waiting.
Another factor to pair with risk appetite is opportunity cost. By delaying the end of one experiment by one month to seek 95% significance you are also losing the opportunity to run the next experiment on that page or other pages.
So waiting for that magical significance number can end up costing you more than what you were prepared to risk initially.
This is especially true for high traffic websites. Any delay in implementing your next test could really cost you.
So let’s not get caught up in reaching a certain level of significance for significance’s sake. It’s all about weighing up the risk and acknowledging the opportunity cost that comes with delay.
Sean Blake Blog 