\magnification=1200
\baselineskip=20pt
\nopagenumbers
\font\big=cmr12 scaled \magstep2

\centerline{\bf STANFORD UNIVERSITY}
\centerline{\bf DEPARTMENT OF STATISTICS}
\centerline{\big DEPARTMENTAL SEMINAR}
\bigskip

\baselineskip=12pt
\centerline{4:15 p.m., Tuesday, October 26, 1999}
\centerline{Sequoia Hall Rm. 200}
\centerline{(Cookies at 3:45 in 1st Floor Lounge)}

\bigskip
\baselineskip=15pt
\centerline{\sl Hidetoshi Shimodaira}
\centerline{\sl The Institute of Statistical Mathematics, Tokyo, JAPAN}
\bigskip
\centerline{\bf Assessing the Uncertainty in Model Selection ---
The Confidence Set of Models.}
\centerline{\bf }
\bigskip



ABSTRACT:

We consider multiple comparisons of log-likelihoods, or AIC values, to take
account of the multiplicity of testings in selection of nonnested models. The
result is obtained as a set of equally good models, which are not significantly
worse than the maximum  likelihood model; i.e., a confidence set of models
(Shimodaira 1993, 1998).  This is quite different from another stream of
testing of nonnested models such as the Cox test or the LR test against the
full model, since the former is testing as to which model is better than the
other, while the latter is to find the correct one.  When comparing just two
models, our method reduces to the test of Linhart (1988), Kishino and Hasegawa
(1989), and Vuong (1989); which is known as the K-H test, and is widely used in
inferring the tree topology of molecular phylogeny. The K-H test can give
over-confidence to a wrong tree topology (or model in general), because the
selection bias in the log-likelihood difference is overlooked in it. Our method
automatically corrects the selection bias, and a numerical example of
Shimodaira and Hasegawa (1999) shows that some controversy over incompatible
biological hypotheses of phylogeny is due to the over-confidence. Recently the
method is applied to the evolution of Maori-cicada in Buckley et al (submitted)
as well.  The uncertainty in topology selection is partially due to the
misspecification of the DNA substitution process --- this often let the LR test
reject all the possible hypotheses!  We touch on a graphical diagnostic method
for this problem.  We also discuss a connection between the method of "The
Problem of Regions" (Efron and Tibshirani 1998) and our method of multiple
comparisons of log-likelihoods. The numerical examples also include the
variable selection of regression analysis.


KEYWORDS: bootstrap bias correction, confidence set of models; information
criterion; model selection; molecular evolution; multiple comparisons;
multiplicity of testings; nonnested models; selection bias; variable
selection
in regression.


\vfill

\bye