A Comparison of Different ROC Measures for Ordinal Regression.pdf

A Comparison of Different ROC Measures for Ordinal Regression Willem Waegeman Willem.Waegeman@UGent.be Department of Electrical Energy, Systems and Automation, Ghent University, Technologiepark 913, B-9052 Ghent, Belgium Bernard De Baets Bernard.DeBaets@UGent.be Department of Applied Mathematics, Biometrics and Process Control, Ghent University, Coupure links 653, B-9000 Ghent, Belgium Luc Boullart Luc.Boullart@UGent.be Department of Electrical Energy, Systems and Automation, Ghent University, Technologiepark 913, B-9052 Ghent, Belgium Abstract Ordinal regression learning has characteris- tics of both multi-class classification and met- ric regression because labels take ordered, discrete values. In applications of ordinal re- gression, the misclassification cost among the classes often differs and with different mis- classification costs the common performance measures are not appropriate. Therefore we extend ROC analysis principles to ordinal re- gression. We derive an exact expression for the volume under the ROC surface (VUS) spanned by the true positive rates for each class and show its interpretation as the prob- ability that a randomly drawn sequence with one object of each class is correctly ranked. Because the computation of V US has a huge time complexity, we also propose three ap- proximations to this measure. Furthermore, the properties of VUS and its relationship with the approximations are analyzed by sim- ulation. The results demonstrate that opti- mizing various measures will lead to different models. 1. Introduction In multi-class classification labels are picked from a set of unordered categories. In metric regression labels might take continuous values. Ordinal regression can Appearing in Proceedings of the ICML 2006 workshop on ROC Analysis in Machine Learning, Pittsburgh, USA, 2006. Copyright 2006 by the author(s)/owner(s). be located in between these learning problems because here labels are chosen from a set of ordered categories. Applications