Contents
Do you need to correct for multiple comparisons?
Some statisticians recommend never correcting for multiple comparisons while analyzing data (1,2). Instead report all of the individual P values and confidence intervals, and make it clear that no mathematical correction was made for multiple comparisons. This approach requires that all comparisons be reported.
How can the chance of committing a Type I error be reduced when performing multiple comparisons?
A Type I error is when we reject a true null hypothesis. Lower values of α make it harder to reject the null hypothesis, so choosing lower values for α can reduce the probability of a Type I error.
How do you correct multiple Anovas?
To correct for multiple comparisons of the main ANOVA P values in Prism, you should copy all the P values from the ANOVA results table and paste into one column of a Column table. If you did a three-way ANOVA, you would copy-paste seven P values into one new column.
What happens when you perform multiple t tests?
Every time you conduct a t-test there is a chance that you will make a Type I error. By running two t-tests on the same data you will have increased your chance of “making a mistake” to 10%. The formula for determining the new error rate for multiple t-tests is not as simple as multiplying 5% by the number of tests.
Why do we do multiple testing correction?
Multiple testing correction adjusts the individual p-value for each gene to keep the overall error rate (or false positive rate) to less than or equal to the user-specified p-value cutoff or error rate.
Which is the best test for multiple testing?
Multiple hypothesis testing is a major issue in genome-wide association studies (GWAS), which often analyze millions of markers. The permutation test is considered to be the gold standard in multiple testing correction as it accurately takes into account the correlation structure of the genome.
How to develop a strategy for multiple testing?
Guidelines for Developing a Strategy for Multiple Testing Delineate separate outcome domains in the study protocols. Outcome domains should be delineated using theory or a conceptual framework that relates the program or intervention to the outcomes. Define confirmatory and exploratory analysis components prior to data analysis.
Why is multiple testing less of a problem?
Multiple testing is less of a problem if studies limit the number of contrasts for analysis. Sharply focusing research questions on one or a few outcomes and on a small number of target groups diminishes the chance of finding impacts where none exist.
How to correct for multiple comparisons within a single study?
If you trim a multivariate model using AIC, which focuses on overall performance of the model, often a few of the surviving parameters have p > 0.05. Along these lines, I had a question about how best to correct for multiple comparisons when performing a few multiple regressions within a single study.