Short Course

Morning session:

by Haiyan Xu ( Johnson & Johnson), Dong Xi (Norvatis), Jason C. Hsu (The Ohio State University) and Frank Bretz (Norvatis)

by Yufeng Liu (UNC) and Haoda Fu (Eli Lilly)

Afternoon session:

by Christopher Jennison (University of Bath)

by Cyrus Mehta (Cytel Inc. ), Lingyun Liu (Cytel Inc.) and Ajit C. Tamhane (Northwestern University)

Fundamentals of Multiple Testing and Graphical Approaches to Multiple Testing Problems

Haiyan Xu ( Johnson & Johnson) , Dong Xi (Norvatis), Jason C. Hsu (The Ohio State University), Frank Bretz (Norvatis)

Two main principles provide the foundation of multiple testing: Closed testing and partitioning. Most multiple comparison methods can be derived and their validity can be proven using these two principles. In this course we show how they are connected using several examples. Starting with realistic numerical examples, the first and conceptual part of this short course will show that the traditional methods of Holm, Hochberg, and Hommel are special cases of closed testing and partitioning. To give insight into how the partitioning principle simplifies challenging problems, we show how Hsu and Berger (1999) formulated the problem of testing multiple doses in a pre-determined step-wise fashion to guarantee decision-making following a pre-specified path. We then show how Liu and Hsu (2009) applied the same path partitioning principle to simplify testing with multiple paths, such as testing for efficacy in multiple doses in combination with multiple endpoints. To conclude the first part of the course, we show how the gatekeeping method of Xu et al (2009), the graphical approach of Bretz et al (2011), and the partition testing principle of Liu and Hsu (2009) coincide and rely on the same testing principles.

The second part of this short course will be on the graphical approach’s flexible and transparent implementation of multiple testing. Using graphical approaches (Bretz et al, 2009), one can easily construct and explore different test strategies and thus tailor the test procedure to the given study objectives. The resulting multiple test procedures are represented by directed, weighted graphs, where each node corresponds to an elementary hypothesis, together with a simple algorithm to generate such graphs while sequentially testing the individual hypotheses. We also present one case study to illustrate how the approach can be used in clinical practice. The presented methods will be illustrated using the graphical user interface from the gMCP package in R, which is freely available on CRAN.


  1. Hsu, Jason C. and Berger, Roger L. (1999). Stepwise Confidence Intervals without Multiplicity Adjustment for Dose-Response and Toxicity Studies. Journal of the American Statistical Association, 94: 468-482.
  2. Liu, Yi and Hsu, Jason C. (2009). Testing for efficacy in primary and secondary endpoints by partitioning decision paths. Journal of the American Statistical Association, 104: 1661-1670.
  3. Xu, Haiyan and Nuamah, Isaac and Liu, Jingyi and Lim, Pilar and Sampson, Allan. (2009). A Dunnett-Bonferroni-based parallel gatekeeping procedure for dose-response clinical trials with multiple endpoints. Pharmaceutical statistics, 8: 301-316.
  4. Bretz, Frank and Posch, Martin and Glimm, Ekkehard and Klinglmueller, Florian and Maurer, Willi and Rohmeyer, Kornelius. (2011). Graphical approaches for multiple comparison procedures using weighted Bonferroni, Simes, or parametric tests. Biometrical Journal, 53: 894-913.
  5. Bretz, Frank and Maurer, Willi and Brannath, Werner and Posch, Martin. (2009). A graphical approach to sequentially rejective multiple test procedures. Statistics in medicine, 28: 586-604.

Artificial Intelligence, Machine Learning, and Precision Medicine

Yufeng Liu (UNC) and Haoda Fu (Eli Lilly)

This half-day short course will provide an overview of statistical machine learning, and artificial intelligence techniques with applications to the precision medicine, in particular to deriving optimal individualized treatment strategies for personalized medicine. This short course will cover both treatment selection and treatment transition. The treatment selection framework is based on outcome weighted classification. We will cover logistic regression, support vector machine (SVM), ?-learning, robust SVM, and angle based classifiers for multi-category learning, and we will show how to modify these classification methods into outcome weighted learning algorithms for personalized medicine. The second part of short course will also cover the treatment transition. We will provide an introduction on reinforcement learning techniques. Algorithms, including dynamic programming for Markov Decision Process, temporal difference learning, SARSA, Q-Learning algorithms, actor-critic methods, will be covered. We will discuss on how to use these methods for developing optimal treatment transition strategies. The techniques discussed will be demonstrated in R. This course is intended for graduate students who have some knowledge of statistics and want to be introduced to statistical machine learning, or practioners who would like to apply statistical machine learning techniques to their problems on personalized medicine and other biomedical applications.

Multiple Hypothesis Testing in Group Sequential and Adaptive Clinical Trials

Christopher Jennison (University of Bath)

The course will introduce group sequential designs and their applications, including error-spending tests and inference for a secondary endpoint on termination of a group sequential trial. The complexity of the testing problem increases as more null hypotheses are tested during or after a sequential trial: we shall describe the general framework for such trials that combines the graphical approach to multiple hypothesis testing with group sequential tests of individual hypotheses.

Adaptive designs allow mid-course modification of a trial while still protecting the type I error rate. Possible modifications include: enrichment designs, which shift their focus to a subset of the initial study population; seamless designs, which combine treatment selection and testing in a single trial; multi-arm Phase III trials which may drop treatments at interim analyses or stop early for a positive outcome. We shall describe the general approach to creating adaptive designs by combining close testing procedures and combination tests, and illustrate these ideas in a case study of a Phase III trial with treatment selection and a survival endpoint.

Trial Designs with Multiple Treatments and Multiple Endpoints Using East®

Cyrus Mehta, Cytel Inc.
Lingyun Liu, Cytel Inc.
Ajit C. Tamhane, Northwestern University

Modern clinical trials are often designed to address multiple clinical questions which need multiplicity adjustments to ensure strong type I error control. Commonly encountered sources of multiplicities include multiple treatments, multiple endpoints, interim analyses and subgroup analyses. This workshop will cover three types of trial designs: (1) testing multiple endpoints with gatekeeping procedures, (2) compare multiple treatments/doses to a common control in group sequential design (MAMS), (3) seamless adaptive design with treatment selection and sample size re-estimation using p-value combination approach. These methods will be illustrated with the help of the East software with real clinical trial examples.