Outcomes of the equivalence of adaptive ridge with least absolute shrinkage.pdf

Outcomes of the Equivalence of Adaptive Ridge with Least Absolute Shrinkage Yves Grandvalet Ste?phane Canu Heudiasyc, UMR CNRS 6599, Universite? de Technologie de Compie?gne, BP 20.529, 60205 Compie?gne cedex, France Yves.Grandvalet@hds.utc.fr Abstract Adaptive Ridge is a special form of Ridge regression, balancing the quadratic penalization on each parameter of the model. It was shown to be equivalent to Lasso (least absolute shrinkage and selection operator), in the sense that both procedures produce the same estimate. Lasso can thus be viewed as a particular quadratic penalizer. From this observation, we derive a fixed point algorithm to compute the Lasso solution. The analogy provides also a new hyper-parameter for tun- ing effectively the model complexity. We finally present a series of possi- ble extensions of lasso performing sparse regression in kernel smoothing, additive modeling and neural net training. 1 INTRODUCTION In supervised learning, we have a set of explicative variables x from which we wish to pre- dict a response variable y. To solve this problem, a learning algorithm is used to produce a predictor bf (x) from a learning set s` = f(xi; yi)gi?=1 of examples. The goal of prediction may be: 1) to provide an accurate prediction of future responses, accuracy being measured by a user-defined loss function; 2) to quantify the effect of each explicative variable in the response; 3) to better understand the underlying phenomenon. Penalization is extensively used in learning algorithms. It decreases the predictor variability to improve the prediction accuracy. It is also expected to produce models with few non-zero coefficients if interpretation is planned. Ridge regression and Subset Selection are the two main penalization procedures. The for- mer is stable, but does not shrink parameters to zero, the latter gives simple models, but is unstable [1]. These observations motivated the search for new penalization techniques such as Garrotte, Non-Negative Gar