Contents
What is minimum detectable difference?
The minimum detectable difference (MDD) is a measure of the difference between the means of a treatment and the control that must exist to detect a statistically significant effect. It is a measure at a defined level of probability and a given variability of the data.
What is minimum detectable effect AB testing?
The minimum detectable effect represents the relative minimum improvement over the baseline that you’re willing to detect in an experiment, to a certain degree of statistical significance. It can help you figure out the likely relationship between impact and effort – or cost and potential value – for your experiment.
How is MDE calculated?
Estimate the desired conversion rate lift So, you have to configure an experiment in such a way that it declares the winner when the conversion rate difference is at least 22% – 20% = 2%. In this example, 2% of the 20% baseline conversion rate is 10% – this is your estimated MDE for the experiment.
How to calculate effect sizes in dependent testing?
Effect sizes can be obtained by using the tests statistics from hypothesis tests, like Student t tests, as well. In case of independent samples, the result is essentially the same as in effect size calculation #2. Dependent testing usually yields a higher power, because the interconnection between data points of different measurements are kept.
Which is more informative, the minimum detectable effect or the MDE?
The resulting value, which is known as the minimum detectable effect (MDE), is more informative, but it also has shortcomings for the post hoc interpretation of nonsignificant results. Most importantly, the MDE is independent of the estimated effect size.
When to use a nonparametric test in statistics?
Parametric tests involve specific probability distributions (e.g., the normal distribution) and the tests involve estimation of the key parameters of that distribution (e.g., the mean or difference in means) from the sample data.
When is a nonparametric test robust to the central limit?
Tests are robust in the presence of violations of the normality assumption when the sample size is large based on the Central Limit Theorem (see page 11 in the module on Probability).