Bhavik Patel

Another way to think about this is: what is your appetite for risk? i.e. how severe is being wrong? If the severity is low, you can probably afford to take a bigger risk and increase your significance level from 5% to 10% or even 15%. The benefit of this is that it speeds up your time to experiment and make decisions more quickly. In a world where most business decisions are made without any insights, even a significance level of 0.15 is a high level of confidence.

The problem is that in most cases (especially in online experimentation) previous studies, literature reviews or assumptions rarely exist and if they do, does it make sense to test again yourself? The problem with this approach is that it's so counter intuitive. It's easier to estimate what sample size is available to you and then figure out from there what the minimum detectable effect is. If the effect from the estimated sample is too high than what you believe, then you can decide wether or not you have the funds/resources available to gain a bigger sample or run the experiment long - that might not be possible. Alternatively if the detectable effect is low, you can save time/money/resources by reducing the sample size.

This point probably does more harm than good for anyone who is new to this space. In essence your experimentation tool (assuming for online tests) will handle the randomisation element for you and control for such noise. The main thing you need to worry about is your unit of randomisation, for example should it be sessions or users and ensuring you have enough of a sample to control for that noise... especially when you're working with continuous metrics (such as AOV, RPV etc...)

This is somewhat backwards, and it's a common misconception that you need to calculate sample size. You actually know what type of sample size you have for any given period/segment - or at least an estimate of it. What you don't know is what the expected effect size is. So instead of calculating sample size, you're better off calculating the minimum detectable effect with the traffic you have available in that period and determining whether or not that is realistic. You might not need to run the test for as long as you think or you might need to run for longer. I've built a calculator to do this (for free) here: https://www.causl.co.uk/causl-analytics

This is a really good point. Most people assume that a classic A/B test (randomised controlled trial) is your only option. Whilst it's the best option to understand the causal relationship between two variables, it's not the only one. Especially if you have limitations such as low traffic volumes.