A/B Testing: Mazher Khan - IIT (BHU) - B.Tech (DR-2)
A/B Testing: Mazher Khan - IIT (BHU) - B.Tech (DR-2)
@khan.the.analsyt
Statistical signifiance is a term used to indicate whether the results of a study are likely due to
chance or are actually meaningful. In the context of A/B testing, statistical significance is used
to determine whether the observed difference in conversion rates between two groups
@khan.the.analsyt
6. Draw conclusions: If the null hypothesis is rejected, you can conclude that the new
design performs better than the current design in terms of conversion rate. If the null
hypothesis is not rejected, you cannot conclude that there is a difference in conversion
rates between the two designs.
7. L2Q Q2B B2I I2P L2B L2P
8. 57.80% -45.80% -35.50% 130.60% -14.40% 27.20%
Type 1 error (False Positive): When you reject a true null hypothesis. In other words, you
conclude that there is a significant difference between the two groups when, in reality, there is
no difference.
Type 2 error (False Negative): When you fail to reject a false null hypothesis. In other words, you
conclude that there is no significant difference between the two groups when, in reality, there is
a difference.
The relationship between Type 1 and Type 2 errors is inverse. As the probability of making a
Type 1 error decreases, the probability of making a Type 2 error increases, and vice versa.
Therefore, it's important to balance the risk of these two errors in hypothesis testing.
1. Which Type error is more important: Understanding the consequences of each type of
error can help you determine the appropriate balance between them.
2. Consider the effect size: The effect size is the magnitude of the difference between the
two groups being tested. Larger effect sizes are easier to detect, which reduces the risk
of both Type 1 and Type 2 errors. If you have a small effect size, you may need a larger
sample size to achieve the desired balance between Type 1 and Type 2 errors.
3. Adjust the significance level: The significance level (alpha) determines the probability of
making a Type 1 error. A lower significance level reduces the risk of Type 1 errors but
increases the risk of Type 2 errors. Conversely, a higher significance level increases the
risk of Type 1 errors but decreases the risk of Type 2 errors. You can adjust the
significance level based on the importance of each type of error and the consequences
of making them.
4. Increase the sample size: Increasing the sample size can reduce the risk of Type 2
errors, but it also increases the cost and time required to conduct the experiment. The
optimal sample size depends on the effect size, the significance level, and the desired
balance between Type 1 and Type 2 errors.
1. Review the data: To see if there are any patterns or trends or anomalies
2. Re-evaluate your hypothesis:
3. Extend the test:
4. Consult with experts:
@khan.the.analsyt
How long A/B test should be run
1. The length of time an A/B test should be conducted depends on several factors:
a. Sample size, the effect size, and the amount of traffic on your website.
2. As a general rule, should run an A/B test for long enough to collect a sufficient amount
of data to ensure that the results are statistically significant and is not by chance.
3. If you stop the test too soon, you may end up with inconclusive or unreliable results. On
the other hand, if you run the test for too long, you may end up wasting resources and
delaying the implementation of changes that could drive improvements in your business
metrics.
4. Once you have determined the minimum sample size, you can estimate how long it will
take to collect that amount of data based on your website traffic and conversion rates.
5. In general, a minimum test duration of two weeks is recommended to ensure that the
results are reliable. However, the optimal test duration will depend on your specific
circumstances, so it's important to use a sample size calculator and consult with experts
if you're unsure.
What are some common pitfalls to avoid when running an A/B test?
@khan.the.analsyt