Contents
Why is lasso not a solution for all problems?
However, it is not a solution for all problems. If the underlying model is linear (Y = Xβ + ϵ), non-zero λ leads to bias in the lasso solution (E (βˡᵃˢˢᵒ) ≠ β but the estimator has lower variance than MLE) — therefore it cannot achieve estimation and selection consistency simultaneously even for simple cases.
How does lasso penalty help in linear regression?
Lasso penalty helps in achieving this goal partially. We observe that the OLS solution does not exist (may not be unique) if the covariance matrix is non-invertible. Analytical solution does not exist for this minimization. Also gradient descent is not guaranteed to converge on this loss function even though it is convex.
What can lasso regularization be used for in general?
Lasso regularization can be extended to a wide variety of objective functions such as those for generalized linear models, generalized estimating equations, proportional hazards models, and M-estimators in general, in the obvious way.
How does lasso penalty work in deep learning?
Lasso penalty creates sparsity in coefficients by driving some of the coefficient to 0. This applies to linear regression and fully-connected layers in deep neural networks. Therefore lasso penalty can reduce the complexity of deep learning models for suitable values of λ. However, it is not a solution for all problems.
When to use Lasso regression in a study?
Lasso regression tends to set a lot of β to zero. It is useful in studies where you have many potential x but only a few observations (say, genetics) or where you don’t have a strong theoretical reason to believe that all x matter. I think the most common approach is actually to do both lasso and ridge at the same time (the elastic nets approach).
What makes the objective infinite in Ridge Lasso regression?
Because of infinite weightage on square of coefficients, anything less than zero will make the objective infinite. The magnitude of α will decide the weightage given to different parts of objective. The coefficients will be somewhere between 0 and ones for simple linear regression.