A Study of CrossValidation and Bootstrap for Accuracy Estimation and Model Selection.pdfVIP

App ears in the International Joint Conference on Articial Intelligence (IJCAI), 1995 A Study of Cross-Validation and Bo otstrap for Accuracy Estimation and Mo del Selection Ron Kohavi Computer Science Department Stanford University Stanford, CA. 94305 ronnyk@CS.Stanford .EDU http://robot /~ronnyk Abstract for low variance, assuming the bias aects all classiers similarly ( e.g., estimates are 5% p essimistic). We review accuracy estimation metho ds and In this pap er we explain some of the assumptions made compare the two most common metho ds: cross- by the dierent estimation metho ds, and present con- validation and b o otstrap. Recent exp erimen- crete examples where each metho d fails. While it is tal results on articial data and theoretical re- known that no accuracy estimation can b e correct all sults in restricted settings have shown that for the time (Wolp ert 1994 b, Schaer 1994), we are inter- selecting a go o d classier from a set of classi- ested in identifying a metho d that is well suited for the ers (mo del selection), ten-fold cross-validation biases and trends in typical real world datasets. may b e b etter than the more exp ensive leave- Recent results, b oth theoretical and exp erime