Institution: Educational Testing Service
Street: Rosedale Road
City: Princeton, NJ
Authors: Charles Lewis and Dorothy T. Thayer
Title: Multiple Inferences for Random Effects
Abstract: Shaffer (1999) recently studied the performance of several multiple comparison procedures (MCPs) in what is essentially a random-effects environment (population means sampled from a Normal distribution). Specifically, she found that, once it had been adjusted to provide weak familywise Type I error probability (FWE) control, a modification of Duncan?s Bayesian MCP (Duncan, 1965; Waller and Duncan, 1969) performed similarly in this environment to Benjamini and Hochberg?s (1995) procedure for controlling the false discovery rate (FDR). We reformulate Duncan?s Bayesian procedure in a sampling-theory random-effects framework, but using a different loss function. Our loss function is chosen so that, when its (posterior) expectation is minimized, the resulting procedure provides explicit per-comparison control of the probability of ?wrong-sign? declarations in the spirit of, for instance, Jones and Tukey (2000). We further adopt the ?wrong-sign? modification of the FDR proposed by Williams, Jones and Tukey (1999) and show that it, when averaged over all sets of population means, is also controlled by the reformulated Bayesian procedure for any value of the between-means variance, assuming the latter is known. This fact provides theoretical support for Shaffer?s simulation results. Finally, we propose a minor variation of Shaffer?s modified Duncan procedure (using, as they did, a sample estimate of the between-means variance) which guarantees control of the (?wrong-sign?) FDR. Simulations of pairwise comparisons for random effects demonstrate similar (but slightly superior) power, expressed as the average number of correct sign declarations, for this procedure compared to that of Benjamini and Hochberg.
References: Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57, 289-300.
Duncan, D. B. (1965). A Bayesian approach to multiple comparisons. Technometrics, 7, 171-222.
Jones, L. V., & Tukey, J. W. (2000). A sensible formulation of the significance test. Psychological Methods, 5, 411-414.
Shaffer, J. P. (1999). A semi-Bayesian study of Duncans Bayesian multiple comparison procedure. Journal of Statistical Planning and Inference, 82, 197-213.
Waller, R. A., & Duncan, D. B. (1969). A Bayes rule for symmetric multiple comparisons problems. Journal of the American Statistical Association, 64, 1484-1503.
Williams, V. S. L., Jones, L.V., & Tukey, J. W. (1999). Controlling error in multiple comparisons, with examples from state-to-state differences in educational achievement. Journal of Educational and Behavioral Statistics, 24, 42-69.