Why do we use the sample conversion rate?
The sample conversion rate is the control conversion rate while conducting the test. We use the sample conversion rate to draw conclusions about the population conversion rate. This is how the statistics work: you draw conclusions from the population based on what you see for your sample.
When to raise the sample size for a / B test?
“Raising the sample size when the result looks promising, where the definition of promising depends on the amount of knowledge gathered so far, guarantees the protection of the type I error rate”. What if my website or campaign does not receive enough visitors to match the required A/B test sample size?
When to specify a variable in experimental design?
Specify the variable. You want to make sure this is something that you can measure, and make sure that you control for all other factors too.
Is there a difference between the two conversion rates?
There is no difference between the two conversion rates of the control and the variation (they are EXACTLY the same!) There is a difference between the two conversion rates but you don’t have enough sample size (power) to detect it. The first case is very rare since the two conversion rates are usually different.
How to calculate Sample Size for an AB test?
Another way to calculate the sample size for an AB test is by using the confidence interval. From the definition, the confidence interval is a type of interval estimate that contains the true values of our parameter of interest with a given probability.
When to use t distribution instead of normal distribution?
The T-distribution is used instead of the normal distribution when you have small samples (usually in practice less than 30). The larger the size of your sample, the more the t-distribution looks like the normal one.
Which is an example of an AB test?
An example of this: we assume a coin is fair. Can we determine if this assumption is reasonable if we flip the coin 100 times? AB testing is taking two randomized samples from a population, a Control and a Variant sample, and determining if the difference between those two samples are significant.