Institution: Friedrich Schiller University Jena, Institute of Medical Statistics, Computer Sciences and Documentation
Street: Jahnstr. 3
Phone: 49 03641 934136
Fax: 49 03641 933200
Authors: Manfred Horn, Friedrich Schiller University Jena, Germany Charles W. Dunnett, McMaster University Hamilton, Canada
Title: Power and sample size comparisons of some FWE and FDR controlling step-down and step-up procedures in the many-one case
Abstract: Assume X0, X1, . . .,Xk are normally distributed random variables with the expectations µ0, µ1, . . .,µk and common variance $\sigma^2$. X0 represents a control against which k treatments are to be compared. There exist several step-down (SD) and step-up (SU) procedures for testing the hypotheses Hi: µi=µ0. Some of them control the familywise error rate (FWE), others the false discovery rate (FDR). The latter were only recently developed with the claim to achieve higher powers than with FWE controlling methods. Our task is to compare the powers of the FWE and FDR controlling procedures under the assumption of normality, and to investigate how much smaller the sample sizes can be with FDR control compared with FWE control in order to reject the hypotheses Hi (i=1,...,k) with specified power 1-ß. We consider configurations where $m \leq k$ hypotheses are false and calculate the probability to reject at least t of m false hypotheses. Our computer program permits to determine the least favorable configuration (LFC) of step-down (SD) and step-up (SU) procedures, see Dunnett, Horn, Vollandt (2001). Our numerical results show that the FDR controlling SD method of Troendle (2000) which utilizes the multivariate t-distribution of the test statistics strongly dominates the other FWE and FDR controlling methods concerning the all-pairs power. However, among the remaining methods no one dominates the others over all configurations. Concerning the per-pair or any-pair power, the established FWE controlling procedures are superior for most configurations. In addition to the power values, we calculated sample size ratios with the sample size for the FWE controlling SD procedure of Dunnett and Tamhane (1991) in the denominator. Interestingly, with increasing k this ratio is distinctly decreasing for the FDR controlling methods of Troendle (2000), Benjamini, Hochberg (1995) and Sarkar (2001) which means that the advantage of these three methods over the SD procedure of Dunnett and Tamhane (1991) becomes the greater the greater k is. This is an important and desirable property which the other methods do not have.
References: 1. Benjamini, Y., Hochberg, Y. (1995): Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. R. Statist. Soc. B, 289-300. 2. Dunnett, C.W., Tamhane, A.C. (1991): Step-own multiple tests for comparing treatments with a control in unbalanced one-way layouts. Statist. Med. 10, 939-947. 3. Dunnett, C.W., Horn, M., Vollandt, R. (2001): Sample size determination in step-down and step-up multiple tests for comparing treatments with a control. JSPI 97, 367-384. 4. Sarkar, S.K. (2001): Some results on false discovery rate in stepwise multiple testing procedures. Technical Report #01-01, September 24, 2001. 5. Troendle, J (2000): Stepwise normal theory multiple test procedures controlling the false discovery rate. JSPI 84, 139-158.