Every ML practitioner hits the same wall: your model looks brilliant on training data, then falls apart on real data — or it never learned anything useful in the first place. These are overfitting and underfitting, and understanding them deeply is the difference between models that work in the lab and models that work in production.
What is Overfitting?
Overfitting happens when your model learns the training data too well — including its noise, outliers, and random quirks that have nothing to do with the underlying pattern. The model memorises instead of generalising.
Classic symptom
Training accuracy: 99% → Validation accuracy: 72%. That gap is the fingerprint of overfitting.
Think of a student who memorises every answer in the textbook word-for-word but cannot solve a problem that is phrased differently. They have overfit the textbook.
What is Underfitting?
Underfitting is the opposite: the model is too simple to capture the patterns in the data. It performs poorly on both training and validation sets. This usually means your model lacks the capacity (too few parameters, too shallow, too much regularisation) to learn what it needs to.
Classic symptom
Training accuracy: 61% → Validation accuracy: 60%. Low on both sides — the model has not learned anything meaningful.
The Bias–Variance Tradeoff
These two failure modes map directly onto a foundational ML concept: bias and variance.
High Bias = Underfitting
The model has oversimplified assumptions. It consistently misses the target — not due to randomness, but due to a fundamentally wrong model structure.
High Variance = Overfitting
The model is too sensitive to the training data. Small changes in input produce wildly different predictions. It has learned noise as signal.
The goal is to find the sweet spot — low bias and low variance — which is the region where your model generalises well to unseen data.
How to Detect Each Problem
The most reliable diagnostic tool is the learning curve — plotting training and validation loss (or accuracy) against the number of training examples or epochs.
import numpy as np
import matplotlib.pyplot as plt
from sklearn.model_selection import learning_curve
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier(n_estimators=100, random_state=42)
train_sizes, train_scores, val_scores = learning_curve(
model, X, y,
train_sizes=np.linspace(0.1, 1.0, 10),
cv=5, scoring='accuracy'
)
plt.figure(figsize=(8, 5))
plt.plot(train_sizes, train_scores.mean(axis=1), label='Training accuracy')
plt.plot(train_sizes, val_scores.mean(axis=1), label='Validation accuracy')
plt.xlabel('Training set size')
plt.ylabel('Accuracy')
plt.legend()
plt.title('Learning Curve')
plt.tight_layout()
plt.show()
# Reading the curve:
# Large gap between lines → Overfitting (high variance)
# Both lines low and converging → Underfitting (high bias)5 Ways to Fix Overfitting
Get more training data
More data is often the single most powerful fix. When a model has more diverse examples, it cannot memorise patterns specific to a small sample.
Regularisation (L1 / L2)
Regularisation adds a penalty to the loss function for large weights, forcing the model to stay simpler.
from sklearn.linear_model import Ridge, Lasso # Ridge = L2 regularisation ridge = Ridge(alpha=1.0) # larger alpha = stronger penalty # Lasso = L1 regularisation (also does feature selection) lasso = Lasso(alpha=0.1)
Dropout (neural networks)
Randomly 'drops' neurons during training, preventing any single neuron from becoming overly specialised.
import tensorflow as tf
model = tf.keras.Sequential([
tf.keras.layers.Dense(256, activation='relu'),
tf.keras.layers.Dropout(0.4), # drop 40% of neurons each step
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Dense(1, activation='sigmoid')
])Reduce model complexity
Use a shallower tree, fewer layers, fewer features. A simpler model has less capacity to memorise noise.
from sklearn.tree import DecisionTreeClassifier # Limit depth to prevent memorisation model = DecisionTreeClassifier(max_depth=4, min_samples_leaf=10)
Early stopping
Stop training when validation loss stops improving, before the model has time to overfit.
from tensorflow.keras.callbacks import EarlyStopping
callback = EarlyStopping(
monitor='val_loss',
patience=5, # stop after 5 epochs of no improvement
restore_best_weights=True
)
model.fit(X_train, y_train, validation_split=0.2,
epochs=200, callbacks=[callback])4 Ways to Fix Underfitting
Increase model complexity
Add more layers, increase tree depth, add more neurons. Give the model enough capacity to learn.
Add more features or engineer better ones
If the raw features do not contain enough signal, the model cannot learn the pattern no matter how large it is. Interaction terms, polynomial features, or domain-specific features often help.
from sklearn.preprocessing import PolynomialFeatures poly = PolynomialFeatures(degree=2, include_bias=False) X_poly = poly.fit_transform(X_train) # adds x², x·y, etc.
Reduce regularisation
If you added L1/L2 penalties or strong dropout, try reducing them. You may be constraining the model too much.
Train longer
For deep learning, the model may simply not have been given enough epochs to converge. Monitor validation loss and let it run.
Quick-Reference Cheat Sheet
| Symptom | Diagnosis | Fix |
|---|---|---|
| High train acc, low val acc | Overfitting | More data, regularisation, dropout, early stopping |
| Low train acc, low val acc | Underfitting | Bigger model, more features, less regularisation |
| Both scores low and equal | Underfitting | Wrong model family, noisy labels — rethink the problem |
| Both scores high and equal | Good fit ✓ | Deploy and monitor |
Key Takeaway
Overfitting and underfitting are not bugs — they are signals. A large train–val gap tells you your model has memorised; equal low scores tell you it has not learned. Always plot your learning curves before tuning anything, and let the data tell you which direction to move.