模式识别和机器学习第三章线性回归模型.ppt

Pattern Recognition and Machine Learning Chapter 3: Linear models for regression Linear Basis Function Models (1) Example: Polynomial Curve Fitting Linear Basis Function Models (2) Generally where Áj(x) are known as basis functions. Typically, Á0(x) = 1, so that w0 acts as a bias. In the simplest case, we use linear basis functions : Ád(x) = xd. Linear Basis Function Models (3) Polynomial basis functions: These are global; a small change in x affect all basis functions. Linear Basis Function Models (4) Gaussian basis functions: These are local; a small change in x only affect nearby basis functions. ¹j and s control location and scale (width). Linear Basis Function Models (5) Sigmoidal basis functions: where Also these are local; a small change in x only affect nearby basis functions. ¹j and s control location and scale (slope). Maximum Likelihood and Least Squares (1) Assume observations from a deterministic function with added Gaussian noise: which is the same as saying, Given observed inputs, , and targets, , we obtain the likelihood function Maximum Likelihood and Least Squares (2) Taking the logarithm, we get where is the sum-of-squares error. Computing the gradient and setting it to zero yields Solving for w, we get where Maximum Likelihood and Least Squares (3) Maximum Likelihood and Least Squares (4) Maximizing with respect to the bias, w0, alone, we see that We can also maximize with respect to ¯, giving Geometry of Least Squares Consider S is spanned by . wML minimizes the distance between t and its orthogonal projection on S, i.e. y. N-dimensional M-dimensional Sequential Learning Data items considered one at a time (a.k.a. online learning); use stochastic (sequential) gradient descent: This is known as the least-mean-squares (LMS) algorithm. Issue: how to choose ´? Regularized Least Squares (1) Consider the error function: With the sum-of-squares error function and a quadratic regul