Name: Jin

Firstname: Jiashun

Title: Ph.D Student

Institution: Statistics Department

Street: Stanford University

City: Stanford, CA

Zip-Code: 94305

Country: United States

Phone: (650)725-2236

Fax: (650)725-8977

Email: jjin@stat.stanford.edu

Authors: David L. Donoho and Jiashun Jin

Title: Optimal Adaptivity of Tukeys Higher Criticism

Abstract: {\it Higher Criticism}, or {\it second-level significance testing}, is a multiple comparisons concept mentioned in passing by Tukey (1976). It concerns a situation where there are many independent and unrelated tests of significance. Tukey suggested to compare the fraction of observed significances at a given $\alpha$-level to the expected fraction under the joint null; in fact he suggested to standardize the difference of the two quantities and form a z-score; the resulting z-score tests the significance {\it of the body of significance tests}.

We consider a generalization, where we maximize this z-score over a range of significance levels $0 < \alpha \leq \alpha_0$. We are able to show that the resulting {\it Higher Criticism statistic} is effective at resolving a very subtle testing problem where we are testing whether $n$ normal means are all zero versus the alternative that a very small fraction is nonzero.

The subtlety of this sparse normal means testing problem can be seen from work of Ingster (1999) and Jin (in preparation), who studied such problems in great detail. In their studies, they identified an interesting range of cases where the small fraction of nonzero means is so small that the alternative hypothesis exhibits little noticeable effect on the distribution on the $p$-values either for the bulk of the tests or for the few most highly significant tests. Nevertheless, the fraction of nonzero means is delicately calibrated so that the likelihood ratio test for a precisely-specified alternative would still succeed in separating the two hypotheses.

We show that the higher criticism is successful throughout the same region where the precisely specified likelihood ratio test would succeed, and thus, higher criticism is in a sense optimally adaptive to unknown sparsity and size of the non-null effects. While our theoretical work is largely asymptotic, we provide simulations in finite samples, and suggest some possible applications. \end{abstract}

References: Eicker, F. The asymptotic distribution of the supremum of the standardized empirical process, Ann. Statist. Vol 7,116--138,1979.

Ingster, Yu.I., Some Problems of Hypothesis Testing Leading to Infinitely Divisible Distribution, Math. Methods Statist. Vol 6,1,47--69,1997.

Ingster, Yu.I., Minimax detection of a signal for $l^p n$-balls, Math. Methods Statist. Vol 7,4, 401-428, 1999.

Tukey, John.W. Higher Criticism`, Course notes, Statistics 411 1976,Princeton Universtity.