Holm-Bonferroni method
Encyclopedia
In statistics
, the Holm–Bonferroni method performs more than one hypothesis test
simultaneously. It is named after Sture Holm and Carlo Emilio Bonferroni
.
s and comparing the smallest p-value to α/k. If that p-value is less than α/k, then reject that hypothesis and start all over with the same α and test the remaining k − 1 hypothesis, i.e. order the k − 1 remaining p-values and compare the smallest one to α/(k − 1). Continue doing this until the hypothesis with the smallest p-value cannot be rejected. At that point, stop. None of the remaining hypotheses can be rejected.
Here is an example. Four null hypotheses are tested with α = 0.05. The four unadjusted p-values are 0.01, 0.03, 0.04, and 0.005. The smallest of these is 0.005. Since this is less than 0.05/4, null hypothesis
four is rejected (meaning some alternative hypothesis likely explains the data). The next smallest p-value is 0.01, which is smaller than 0.05/3. So, null hypothesis one is also rejected. The next smallest p-value is 0.03. This is not smaller than 0.05/2, so you fail to reject this hypothesis (meaning you have not seen evidence to conclude an alternative hypothesis is preferable to the level of α = 0.05). As soon as that happens, you stop, and therefore, also fail to reject the remaining hypothesis that has a p-value of 0.04. Therefore, hypotheses one and four are rejected while hypotheses two and three are not rejected. Applying the approximate false discovery rate produces the same result without requiring ordering the p-values, then using different criteria for each test.
. As such, it controls the familywise error rate
for all the k hypotheses at level α in the strong sense. Each intersection is tested using the simple Bonferroni test.
It is also possible to define a weighted version. Let p1,..., pk be the unadjusted p-values and let w1,..., wk
be a set of corresponding positive weights that add to 1. Without loss of generality, assume the p-values and the weights are all ordered such that p1/w1 ≤ p2/w2 ≤ ... ≤ pk/wk. The adjusted p-value for the first hypothesis is q1 = min{1,p1/w1}. Inductively, define the adjusted p-value for hypothesis i by qi = min{1,max{qi−1,(wi + ... + wk)×pi/wi}}. A hypothesis is rejected at level α if and only if its adjusted p-value is less than α. In the earlier example using equal weights, the adjusted p-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.
Statistics
Statistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....
, the Holm–Bonferroni method performs more than one hypothesis test
Statistical hypothesis testing
A statistical hypothesis test is a method of making decisions using data, whether from a controlled experiment or an observational study . In statistics, a result is called statistically significant if it is unlikely to have occurred by chance alone, according to a pre-determined threshold...
simultaneously. It is named after Sture Holm and Carlo Emilio Bonferroni
Carlo Emilio Bonferroni
Carlo Emilio Bonferroni was an Italian mathematician who worked on probability theory. Carlo Emilio Bonferroni was born in Bergamo on 28 January 1892 and died on 18 August 1960 in Firenze . He studied in Torino , held a post as assistant professor at the Turin Polytechnic, and in 1923 took up the...
.
Procedure
Suppose there are k null hypotheses to be tested and the overall type 1 error rate (significance level) is α. Start by ordering the p-valueP-value
In statistical significance testing, the p-value is the probability of obtaining a test statistic at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. One often "rejects the null hypothesis" when the p-value is less than the significance level α ,...
s and comparing the smallest p-value to α/k. If that p-value is less than α/k, then reject that hypothesis and start all over with the same α and test the remaining k − 1 hypothesis, i.e. order the k − 1 remaining p-values and compare the smallest one to α/(k − 1). Continue doing this until the hypothesis with the smallest p-value cannot be rejected. At that point, stop. None of the remaining hypotheses can be rejected.
Here is an example. Four null hypotheses are tested with α = 0.05. The four unadjusted p-values are 0.01, 0.03, 0.04, and 0.005. The smallest of these is 0.005. Since this is less than 0.05/4, null hypothesis
Null hypothesis
The practice of science involves formulating and testing hypotheses, assertions that are capable of being proven false using a test of observed data. The null hypothesis typically corresponds to a general or default position...
four is rejected (meaning some alternative hypothesis likely explains the data). The next smallest p-value is 0.01, which is smaller than 0.05/3. So, null hypothesis one is also rejected. The next smallest p-value is 0.03. This is not smaller than 0.05/2, so you fail to reject this hypothesis (meaning you have not seen evidence to conclude an alternative hypothesis is preferable to the level of α = 0.05). As soon as that happens, you stop, and therefore, also fail to reject the remaining hypothesis that has a p-value of 0.04. Therefore, hypotheses one and four are rejected while hypotheses two and three are not rejected. Applying the approximate false discovery rate produces the same result without requiring ordering the p-values, then using different criteria for each test.
Extensions
The Holm–Bonferroni method is an example of a closed test procedureClosed testing procedure
In statistics, the closed testing procedure is a general method for performing more than one hypothesis test simultaneously.-The closed testing principle:...
. As such, it controls the familywise error rate
Familywise error rate
In statistics, familywise error rate is the probability of making one or more false discoveries, or type I errors among all the hypotheses when performing multiple pairwise tests.-Classification of m hypothesis tests:...
for all the k hypotheses at level α in the strong sense. Each intersection is tested using the simple Bonferroni test.
It is also possible to define a weighted version. Let p1,..., pk be the unadjusted p-values and let w1,..., wk
be a set of corresponding positive weights that add to 1. Without loss of generality, assume the p-values and the weights are all ordered such that p1/w1 ≤ p2/w2 ≤ ... ≤ pk/wk. The adjusted p-value for the first hypothesis is q1 = min{1,p1/w1}. Inductively, define the adjusted p-value for hypothesis i by qi = min{1,max{qi−1,(wi + ... + wk)×pi/wi}}. A hypothesis is rejected at level α if and only if its adjusted p-value is less than α. In the earlier example using equal weights, the adjusted p-values are 0.03, 0.06, 0.06, and 0.02. This is another way to see that using α = 0.05, only hypotheses one and four are rejected by this procedure.