Institution: University of California, Berkeley
Street: Dept. of Statisitcs, 367 Evans Hall # 3860
City: Berkeley, CA
Authors: Juliet Popper Shaffer
Title: Is the null hypothesis ever true? Correspondence between beliefs and multiple comparison procedures
Abstract: There has been a longstanding debate over the usefulness of null hypothesis testing, both in general statistics and in applied fields, particularly psychology. Some contributors feel that such testing is not only inadequate, since estimation may be more informative, but is also deleterious in practice because researchers persist in misinterpreting test results. Others, however, object because they believe the null hypothesis can`t be true. The latter position was advocated by John W. Tukey, who argued that any intervention must have an effect, even if it is small. Jones \& Tukey (2000) point out that, if the truth of the null hypothesis is considered impossible, a test of a null hypothesis concerning a single parameter using a symmetric test statistic can be reinterpreted as a simultaneous test of two directional hypotheses, each at level $\alpha/2$, with resulting maximum Type I error rate of $\alpha/2$.
The history of the testing debate and the arguments advanced will be described, and null hypothesis testing will be compared with directional hypothesis testing in a multiple comparison context, where the relationship between the two approaches is far more complex.
References: Jones \& Tukey (2000), "A sensible formulation of the significance test." Psychological Methods, 5, 411-414.