A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection-英文文献.pdf

App ears in the International Joint Conference on Articial Intelligence IJCAI A Study of CrossValidation and Bo otstrap for Accuracy Estimation and Mo del Selection Ron Kohavi Computer Science Department Stanford University Stanford CA ronnykCSStanford EDU httprobot icsstanfordeduronnyk Abstract for low variance assuming the bias aects all classiers similarlyeg estimates are pessimistic We review accuracy estimation methods and Inthispaperweexplainsomeofthe assumptionsmade comparethetwomostcommonmethods cross by the dierent estimation methods and present con validation and bootstrap Recent experimen crete examples where each method fails While it is tal results on articial data and theoretical re known that no accuracy estimation can be correct all sults in restricted settings have shown that for the time Wolpert b Schaer we are inter selecting a good classier from a set of classi ested in identifying a method that is well suited for the ers modelselection tenfold crossvalidation biases and trends in typical real world datasets may be better than the more expensive leave Recent results both theoretical and experimental oneout crossvalidation We report on a large have shown that it is not always the case that increas scale experimentover half a mil