Contents
When to regularize ridge regression and the lasso?
Regularization: Ridge Regression and the LASSO Agenda Regularization: Ridge Regression and the LASSO Statistics 305: Autumn Quarter 2006/2007 Wednesday, November 29, 2006 Statistics 305: Autumn Quarter 2006/2007 Regularization: Ridge Regression and the LASSO Agenda Agenda 1The Bias-Variance Tradeoff 2Ridge Regression
Which is the solution to PRSS ℓ 2?
The solution to PRSS(βˆ)ℓ 2 is now seen to be: βˆridge λ= (Z ⊤Z+λIp)−1Z y Remember that Z is standardized y is centered Solution is indexed by the tuning parameter λ (more on this later) Inclusion of λ makes problem non-singular even if Z⊤Z is not invertible
Which is the Stata command for component plus residuals?
Stata Commands: cprplot. The component plus residual plot is also known as partial-regression leverage plots, adjusted partial residuals plots or adjusted variable plots. The command “cprplot x” graph each obervation’s residual plus its component predicted from x against values of x.
How to control variance in a ridge regression?
To control variance, we might regularize the coefficients i.e., Might control how large the coefficients grow Might impose the ridge constraint: minimize Xn i=1 (yi−β⊤zi)2s.t. Xp j=1 β2 j≤t ⇔minimize (y −Zβ)⊤(y −Zβ) s.t. Xp j=1 β2 j≤t By convention (very important!):
What is the solution to PRSS ℓ 2 ridge regression?
The ridge solutions The solution to PRSS(βˆ)ℓ 2 is now seen to be: βˆridge λ= (Z ⊤Z+λIp)−1Z y Remember that Z is standardized y is centered
How to calculate prediction error and bias variance?
Prediction error and the bias-variance tradeoff So good estimators should, on average have, small prediction errors Let’s consider the PE at a particular target point z0(see the board for a derivation): PE(z0) = EY|Z=z 0 {(Y −ˆf(Z))2|Z = z0} = σ2 ε+Bias
How is the shrinkage parameter chosen in Lasso?
In lasso or ridge regression, one has to specify a shrinkage parameter, often called by λ or α. This value is often chosen via cross validation by checking a bunch of different values on training data and seeing which yields the best e.g. R 2 on test data.