Contents
When to use the rejection rule for multiple hypothesis tests?
1.2 Multiple Hypotheses When conducting multiple hypothesis tests, if we follow the same rejection rule independently for each test, the resulting probability of making at least one Type I error is substantially higher than the nominal level used for each test, particularly when the number of total tests mis large.
Do you need prior knowledge of multiple hypothesis testing?
If you want to read up on the above, the material is covered by any introductory statistics book, as well as many blog posts, e.g. [MT1] and [MT2]. On the other hand, no prior knowledge about multiple testing is necessary. Also, we ignore all of the intricacies that come with satisfying assumptions of tests.
What are the different types of hypothesis testing?
There are two types of tests: Multiple testing: for every hypothesis, we want to separately test each null hypothesis.
Do you need to test each null hypothesis?
Multiple testing: for every hypothesis, we want to separately test each null hypothesis. While the latter might be more relevant in practice, the former leads to great insight and many methods used for the multiple testing problem can be related back to global hypothesis tests, so let’s look at some interesting results for the global test first.
When do you use non-independent multiple comparisons?
Another experimental design with multiple, non-independent comparisons is when you compare multiple variables between groups, and the variables are correlated with each other within groups.
What happens when you do multiple statistical tests?
This problem, that when you do multiple statistical tests, some fraction will be false positives, has received increasing attention in the last few years.
What is the goal of multiple comparisons corrections?
The goal of multiple comparisons corrections is to reduce the number of false positives, because false positives can be embarrassing, confusing, and cause you and other people to waste your time.