Experimentation

The cost of waiting

Many CRO experts are hardcore believers in statistical significance. And I am too.

It’s fantastic to make data-driven business decisions based on real world evidence. Marketers guess too often.

But where the hardcore reliance on significance can let you down is when it becomes the driving factor in decision making. I see people holding out for a specific level of significance all the time and it distracts from the core mission – incremental growth.

The truth is, there’s nothing magical about 95% statistical significance, 99% or even 75%.

So when it comes time to deciding whether to end a test or keep it running to reach a magical significance level, it’s important to assess risk.

As CEO of New Republique, Nima Yassini writes, there is risk involved at every level of significance.

There’s also a massive cost in waiting.

Another factor to pair with risk appetite is opportunity cost. By delaying the end of one experiment by one month to seek 95% significance you are also losing the opportunity to run the next experiment on that page or other pages.

So waiting for that magical significance number can end up costing you more than what you were prepared to risk initially.

This is especially true for high traffic websites. Any delay in implementing your next test could really cost you.

So let’s not get caught up in reaching a certain level of significance for significance’s sake. It’s all about weighing up the risk and acknowledging the opportunity cost that comes with delay.

Don’t stop

Don’t stop testing.

There’s no such thing as finished anymore. In business, your competitors will fly past you the moment you stop.

Anyone can launch a startup in your industry, this afternoon.

So if you stop moving, you’re going backwards.

The beauty of testing is that you can learn something new each day. It could be as simple as ‘that didn’t work’.

The trick is movement. Momentum.

That’s what keeps you in the game. Not every swing will be a home run. And that’s perfectly acceptable. What’s not acceptable is paralysis.

It’s more than running A/B tests.

It’s more than white lab coats.

More than reaching statistical significance.

It’s a state of mind. A worldview where you’re willing to fail.

A way of thinking that challenges the way you run your business. Experimentation means you’re willing to start from scratch and learn exactly what your customers want.

It trades everything you thought to be true for things you know to be true.

It never stops.

And it’s the only sustainable way to grow.