Locally weighted regression
Contents
Prepare data
load MagicTelescope; [Ntrain,D] = size(Xtrain) Ntest = size(Xtest,1); Ytrain = 2*strcmp('Gamma',Ytrain)-1; Ytest = 2*strcmp('Gamma',Ytest)-1;
Ntrain = 12742 D = 10
Standardize variables
[Xtrain,mu,sigma] = zscore(Xtrain); Xtest = bsxfun(@minus,Xtest,mu); Xtest = bsxfun(@rdivide,Xtest,sigma);
Define kernel
See the kernel ridge regression example (Section_13_2_1) for an explanation.
sigma = 1; kernelfun = @(x1,x2) exp(-sum((x1-x2).^2,2)/2/sigma^2);
Include intercept in the linear model
We include an extra column, both in the training and test data. All elements in this column are set to 1. Later we estimate a vector of linear coefficients for our local model. The first element in this vector is the intercept term.
Xtrain = [ones(Ntrain,1) Xtrain]; Xtest = [ones(Ntest,1) Xtest];
Fit the LWR model
For every observation in the test data, we compute weights as kernel distances to all observations in the training set. We then apply these weights to the training data to obtain Z, the weighted version of X, and v, the weighted version of y. Function bsxfun multiplies every row of Xtrain by the corresponding element of w. Equivalently, we could compute diag(w)*Xtrain, but bsxfun allows us to avoid creating an Ntrain-by-Ntrain matrix diag(w). The .* operator performs element-wise multiplication, and its effect is equivalent to diag(w)*y. Similar to the other examples in this chapter, we find the least-squares solution using the backslash operator.
Yfit = zeros(Ntest,1); for n=1:Ntest q = Xtest(n,:); w = sqrt(kernelfun(Xtrain,repmat(q,Ntrain,1))); Z = bsxfun(@times,Xtrain,w); v = Ytrain.*w; b = Z\v; Yfit(n) = q*b; end
Obtain HIACC
See the kernel ridge regression example (Section_13_2_1) for a definition of HIACC.
[~,tpr] = perfcurve(Ytest,Yfit,1,'xvals',[0.1 0.2]);
acc = mean(tpr)
acc = 0.8171