Lasso Regression
Lasso Regression is a regularization technique used to improve prediction accuracy and implicitly perform feature selection.
The full form of LASSO is:
Least Absolute Shrinkage and Selection Operator
Lasso Regression is based on L1 Regularization.
Lasso Regression in Simple Words
Lasso starts with Linear Regression, but adds a rule that discourages unnecessarily large coefficients. With enough regularization, some coefficients can become exactly zero, so those features stop affecting the prediction.
Start with Features
Use the available input columns.
Fit a Linear Model
Learn one coefficient for each feature.
Add L1 Penalty
Penalize the absolute size of coefficients.
Shrink Some to Zero
Some features may drop out of the model.
A Tiny House-Price Example
Suppose a model starts with four numerical features. After Lasso, the coefficients might look like this:
| Feature | Before Lasso | After Lasso |
|---|---|---|
| House Size | 8.0 | 5.4 |
| Bedrooms | 3.2 | 1.7 |
| Wall Color Code | 0.8 | 0 |
| Random ID Number | -0.5 | 0 |
Important: this is an illustration, not a promise that Lasso will always choose the same features. The result depends on the data and the regularization strength.
Why Lasso Regression is Needed
Machine learning models often suffer from overfitting, too many irrelevant features, complex models, multicollinearity, and poor generalization.
Ordinary Linear Regression estimates a coefficient for every supplied feature. If many predictors are weak, noisy, or highly correlated, the model can become harder to interpret and may generalize poorly. Lasso can help by:
- Penalizing large coefficient magnitudes
- Shrinking coefficients
- Allowing some coefficients to become exactly zero
- Simplifying the model
Beginner Intuition
Imagine you are packing a travel bag.
- You cannot carry everything. So you keep only the most important items.
- Lasso Regression works similarly: it encourages a simpler model by shrinking coefficients, and some may become zero. Which features remain depends on the data and the chosen penalty.
Real-World Example
Suppose we want to predict house prices using: House Size, Location, Bedrooms, Bathrooms, Wall Color, Owner Name, Garden Plants.
Not every feature will necessarily add useful predictive information. Lasso can shrink weak coefficients and, for some settings, make some of them exactly zero.
It may keep:
- House Size
- Location
- Bedrooms
And remove:
- Wall Color
- Garden Plants
- Owner Name
This can simplify the model and may improve generalization, but it should be validated on unseen data.
Visual Understanding of Overfitting
GOOD GENERALIZATION
OVERFITTING
Hierarchy of Regression Algorithms
Linear Regression Recap & Problem
Basic Equation:
Worked Example:
- b₀ = 5, b₁ = 3, x = 4
- y = 5 + 3(4)
- y = 17
Problem with Linear Regression:
With many correlated or noisy predictors, ordinary Linear Regression can produce unstable coefficients and may overfit. Coefficient size also depends on the scale of each feature.
A large number alone is not proof of instability; always consider feature scale.
Core Idea of Lasso Regression
Lasso Regression adds a penalty to the Linear Regression cost function. This penalty shrinks coefficients, and for suitable regularization some coefficients can become exactly zero. A zero coefficient means that feature no longer contributes to the linear prediction.
Lasso Cost Function:
Cost = Prediction Loss + λ × Σ|w|This is a teaching form. Libraries may scale the loss differently; the core idea is the same: data-fit loss plus an L1 penalty on the feature coefficients.
Measures prediction mistakes.
Penalizes coefficient size.
Understanding the L1 Penalty
The L1 penalty uses the absolute value of coefficients.
- Suppose: θ₁ = 3, θ₂ = -5
Step 1 — Absolute values: |3| + |-5| = 3 + 5 = 8
Step 2 — Choose λ = 2: L1 penalty = 2 × 8 = 16
Step 3 — If prediction loss = 20: Total cost = 20 + 16 = 36
Increasing λ makes coefficient size more expensive, so the fitted model is pushed toward smaller—and sometimes zero—coefficients.
Understanding Lambda (λ)
Lambda controls regularization strength.
| Lambda Value | Effect |
|---|---|
| λ = 0 | Normal Linear Regression. |
| Small λ | Small regularization (Weak Penalty, Complex Model). |
| Medium λ | Moderate regularization; whether it is a good choice depends on validation results. |
| Very large λ | Underfitting (Strong Penalty, Simpler Model). |
How Lasso Performs Feature Selection
A distinctive property of Lasso is that it can produce a sparse solution: some coefficients may become exactly zero.
In this illustration, x₂ and x₃ become zero. The real features selected by Lasso depend on the training data and the chosen regularization strength.
Coefficient Shrinking Visualization
Why Lasso Creates Sparse Models
With two coefficients, the L1 constraint has a diamond-shaped boundary. Its sharp corners lie on the axes, which helps explain why Lasso solutions often land at points where one or more coefficients are exactly zero.
/\
/ \
/ \
\ /
\ /
\/Ridge vs Lasso Penalty
Ridge Penalty (L2)
Shrinks coefficients but rarely reaches zero.
Lasso Penalty (L1)
Shrinks coefficients and can make them zero.
| Feature | Ridge | Lasso |
|---|---|---|
| Regularization Type | L2 | L1 |
| Can create exact-zero coefficients | Typically no | Yes |
| Promotes sparsity | No | Yes |
| Useful for feature selection | Not directly | Often |
| Highly correlated features | Often keeps them together | May select one and zero others |
| Interpretability | Keeps all coefficients | Can be simpler when sparse |
Increasing L1 regularization usually increases bias and can reduce variance. The useful tradeoff depends on validation performance.
Bias-Variance Tradeoff based on Lambda (λ)
Multicollinearity in Lasso
What is Multicollinearity?
When features are highly correlated (e.g., House Area and Number of Rooms).
Lasso Behavior with Correlated Features:
With strongly correlated predictors, Lasso may keep one and shrink another to zero. Which one is selected can be unstable when predictors carry very similar information.
- Feature A → Kept
- Feature B → Removed
Cost Function Optimization
Lasso can be optimized in different ways. Scikit-learn's Lasso estimator uses coordinate descent. Gradient-based thinking is still useful for intuition, but the absolute-value penalty has a sharp point at zero, so its update is not simply “ordinary gradient + λ” for every coefficient.
Conceptual Gradient-Style Update (for intuition)
Worked Example
- Current Weight (θ): 10
- Learning Rate (η): 0.1
- Gradient (∂J/∂θ): 4
- Lambda (λ): 1
Step 1: sign(10) = +1
Step 2: combined gradient = 4 + (1 × 1) = 5
Step 3: new weight = 10 - (0.1 × 5) = 9.5
The L1 term adds pressure toward zero. This is a teaching calculation; practical Lasso solvers such as scikit-learn's coordinate-descent solver handle the non-smooth point at zero directly.
Why Feature Scaling is Important
Feature scale matters because coefficient magnitudes depend on the units of the inputs. If one feature is measured in rupees and another in years, the same L1 penalty can affect them very differently. Standardizing numerical features often makes the regularization easier to compare across features.
X_scaled = (X - μ) / σ
// Example scaling calculation
Suppose: X = 50, Mean(μ) = 40, Std Dev(σ) = 5
X_scaled = (50 - 40) / 5
X_scaled = 10 / 5
X_scaled = 2
Python Implementation using Scikit-Learn
import numpy as np
import pandas as pd
from sklearn.linear_model import Lasso, LassoCV
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler# Column 1: Study hours (useful)
# Column 2: Locker number pattern (not useful here)
X = np.array([
[1, 0], [2, 1], [3, 0], [4, 1], [5, 0],
[6, 1], [7, 0], [8, 1], [9, 0], [10, 1]
], dtype=float)
Y = np.array([8, 11, 14, 17, 20, 23, 26, 29, 32, 35], dtype=float)
X_train, X_test, Y_train, Y_test = train_test_split(
X, Y, test_size=0.2, random_state=42
)scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)model = Lasso(alpha=0.5, max_iter=10000)
model.fit(X_train_scaled, Y_train)
predictions = model.predict(X_test_scaled)
print("Coefficients:", np.round(model.coef_, 3))
print("Predictions:", np.round(predictions, 2))import numpy as np
import pandas as pd
from sklearn.linear_model import Lasso, LassoCV
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
# Column 1: Study hours (useful)
# Column 2: Locker number pattern (not useful here)
X = np.array([
[1, 0], [2, 1], [3, 0], [4, 1], [5, 0],
[6, 1], [7, 0], [8, 1], [9, 0], [10, 1]
], dtype=float)
Y = np.array([8, 11, 14, 17, 20, 23, 26, 29, 32, 35], dtype=float)
X_train, X_test, Y_train, Y_test = train_test_split(
X, Y, test_size=0.2, random_state=42
)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
model = Lasso(alpha=0.5, max_iter=10000)
model.fit(X_train_scaled, Y_train)
predictions = model.predict(X_test_scaled)
print("Coefficients:", np.round(model.coef_, 3))
print("Predictions:", np.round(predictions, 2))
# For real projects, choose alpha with cross-validation
cv_model = LassoCV(cv=5, random_state=42, max_iter=10000)
cv_model.fit(X_train_scaled, Y_train)
print("Chosen alpha:", cv_model.alpha_)What should you notice?
The first feature keeps a non-zero coefficient, while the second becomes zero in this tiny example. That is the sparse behaviour Lasso is known for.
The exact values depend on the train/test split, scaling and alpha. For real work, tune alpha using cross-validation rather than assuming one value is always best.
Advantages of Lasso
- Can produce sparse models with some exact-zero coefficients.
- Can reduce overfitting by adding regularization.
- Simplifies models for easier interpretation.
- Can simplify interpretation when many coefficients become zero.
Disadvantages of Lasso
- Sensitive to scaling.
- Can sometimes remove useful features with excessive regularization.
- With highly correlated predictors, the selected feature can be unstable.
Lasso vs Ridge vs Elastic Net
| Feature | Ridge | Lasso | Elastic Net |
|---|---|---|---|
| Regularization | L2 | L1 | L1 + L2 |
| Can create exact-zero coefficients | Typically no | Yes | Yes |
| Promotes sparsity | No | Yes | Yes |
| Correlated features | Often stable | May select one | Can balance L1 + L2 |
| Complexity | Medium | Medium | Higher |
Industry Applications Tree
- Disease Prediction
- Gene Selection
- Medical Diagnostics
- Risk Analysis
- Credit Scoring
- Fraud Detection
- Customer Analytics
- Sales Forecasting
- Advertisement Analysis
End-to-End Lasso Regression Workflow
Common Questions About Lasso Regression
Does Lasso always remove features?
No. Some coefficients become zero only when the data and regularization strength lead to a sparse solution.
Is a bigger alpha always better?
No. Too much regularization can underfit. Choose alpha using validation or cross-validation.
Why should numerical features often be scaled?
Because L1 regularization acts on coefficient magnitudes, and those magnitudes depend on feature units.
What happens with correlated features?
Lasso may keep one correlated predictor and shrink another to zero. Elastic Net can be more stable when groups of predictors are strongly correlated.
Continue Learning
Use these lessons to connect Lasso with the ideas around it:
Final Summary
Lasso Regression is a powerful machine learning regularization technique that improves Linear Regression by penalizing large coefficients using the absolute value of weights (L1 Penalty).
Its most distinctive behaviour is sparsity: for suitable regularization, some coefficients can become exactly zero. This can simplify a model and act as an embedded feature-selection mechanism, but the selected features still need validation.
Most Important Insight to Remember:
"Ridge mainly shrinks coefficients, while Lasso can shrink some coefficients all the way to zero. That makes Lasso useful for sparse models and embedded feature selection—but the chosen features still depend on the data and alpha."