Many CRO experts are hardcore believers in statistical significance. And I am too.
It’s fantastic to make data-driven business decisions based on real world evidence. Marketers guess too often.
But where the hardcore reliance on significance can let you down is when it becomes the driving factor in decision making. I see people holding out for a specific level of significance all the time and it distracts from the core mission – incremental growth.
So when it comes time to deciding whether to end a test or keep it running to reach a magical significance level, it’s important to assess risk.
As CEO of New Republique, Nima Yassini writes, there is risk involved at every level of significance.
There’s also a massive cost in waiting.
Another factor to pair with risk appetite is opportunity cost. By delaying the end of one experiment by one month to seek 95% significance you are also losing the opportunity to run the next experiment on that page or other pages.
So waiting for that magical significance number can end up costing you more than what you were prepared to risk initially.
This is especially true for high traffic websites. Any delay in implementing your next test could really cost you.
So let’s not get caught up in reaching a certain level of significance for significance’s sake. It’s all about weighing up the risk and acknowledging the opportunity cost that comes with delay.
When going into any meeting, presentation or especially a challenging conversation, it really helps to have a framework to structure your thoughts.
Barbara Minto’s ‘Pyramid’ structure is one I’ve just learned about and it’s very effectvel.
It goes like this:
Set the context, environment and go through any supporting data. Generally give the lay of the land or the key facts so far.
What’s gone wrong? How are our plans threatened and what’s the potential impact? Explain how and why it’s got to this point.
There’s nothing better than a question that you already have the answer to. Summarise the complication with a question that prompts a resolution… and then give your pre-prepared answer.
State your proposed solution. Then, explain how you go there. List as many fact-based reasons you can as to why this answer is the best way forward. If someone shoots down one of your reasons, don’t panic! Keep going through the list like a ruthless prosecutor.
At the end, take any questions and address any concerns. If everyone is on board (they should be, because you overwhelmed them with reasons to be), allocate actions so that the solution can come to life.
Even if the four steps above don’t go to plan, at least you had one. Appearing in control is a victory in itself and will help you influence the outcome more often than not.