Underfitting and Overfitting
Machine Learning models are built to learn patterns from data and make predictions on unseen examples. The goal is not simply to score well on the examples used for training, but to learn relationships that remain useful on new data.
Understand First: three possible situations
Underfitting
The model has not learned enough of the useful pattern.
Good Generalization
The model captures useful structure and works well on unseen data.
Overfitting
The model fits training-specific details too closely and generalizes poorly.
A useful model should:
- Learn useful patterns from training data
- Avoid chasing unnecessary noise
- Perform well on unseen data
- Use an appropriate level of complexity for the problem
Underfitting and overfitting can occur with many different algorithms. The practical question is therefore not simply, “Is my training score high?” but, “How different are my training and validation results, and what does that difference tell me?”
A familiar student analogy
| Student Behavior | ML Analogy |
|---|---|
| Did not understand enough of the syllabus | Underfitting |
| Memorized exact practice answers only | Overfitting |
| Understood ideas and can answer new questions | Good Generalization |
Simple visual: too simple, useful, too sensitive
Imagine predicting house prices from house size. The dots are observations. The fitted shape below is only a teaching sketch: the important idea is how model complexity can move from missing the pattern to following the pattern and then to reacting too strongly to individual observations.
Too simple
Useful pattern
Too sensitive
Diagnose It Using Training and Validation Performance
The most useful diagnosis comes from comparing performance on data used for fitting with performance on separate validation or cross-validation data. The exact numbers below are illustrative, but the pattern is what matters.
| Situation | Training Error | Validation Error | What it suggests |
|---|---|---|---|
| Underfitting | 18% | 20% | Both are poor; the model may be too limited. |
| Useful fit | 5% | 7% | Both are reasonably good with a small gap. |
| Overfitting | 1% | 16% | Training is excellent, but unseen-data performance is much worse. |
Generalization gap
The difference between training and validation performance is often called a generalization gap. A large gap can be evidence of overfitting, but always interpret it together with the absolute performance level and the metric being used.
What Is Underfitting?
Underfitting occurs when a model does not capture enough of the useful relationship in the data. A common sign is that performance is poor even on training data, and validation performance is also poor.
Detailed Understanding of Underfitting
Suppose the useful relationship is quadratic, such as y = x². If we restrict the model to a straight line (y = mx + b), that model family may be too limited to represent the curved relationship well.
In that situation, both training and validation error can remain high because the model has not captured enough structure.
Possible Causes of Underfitting
- Model capacity too limited: for example, a straight line for an important nonlinear pattern.
- Insufficient or weak features: the inputs may omit useful information.
- Excessive regularization: overly strong penalties can make a model too constrained.
- Insufficient optimization/training: an iterative model may not have trained enough.
- Feature representation mismatch: the available representation may hide the useful relationship.
Possible Responses
- Try a model that can represent the required structure
- Add better domain-relevant features
- Reduce overly strong regularization
- Train iterative models for an appropriate number of steps
- Check whether preprocessing removed useful signal
Bias connection: underfitting is often associated with high bias—the model's assumptions or limitations make it systematically unable to represent enough of the relationship. This is a useful diagnostic tendency, not a label that must hold identically for every algorithm and dataset.
What Is Overfitting?
Overfitting occurs when a model fits training-specific details so closely that its performance does not carry over well to unseen data. Those details can include noise, outliers, accidental correlations, or patterns that are not stable outside the training sample.
An overfitted model can have extremely strong training performance while performing substantially worse on validation or test data. It does not have to literally memorize every row; the key problem is poor generalization.
Polynomial intuition
Suppose the underlying relationship is fairly smooth. A very high-degree polynomial can be flexible enough to bend around individual noisy observations.
A high degree is not automatically overfitting, and a low degree is not automatically safe. We diagnose the problem from performance on unseen data, not from complexity alone.
Factors That Can Increase Overfitting Risk
- Very flexible model relative to available signal/data.
- Limited training data: random details can have greater influence.
- Many irrelevant/noisy features: the model may discover accidental patterns.
- Weak complexity control: some models benefit from regularization or pruning.
- Excessive iterative training: in some models, validation performance can worsen after continued training.
Possible Responses
- Gather more representative training data when feasible
- Use cross-validation to compare model choices reliably
- Apply appropriate regularization such as L1/L2 where relevant
- Prune or constrain tree complexity
- Use early stopping for iterative models when validation performance stops improving
- Remove irrelevant features using a training-only feature-selection workflow
Important: cross-validation does not itself “remove” overfitting
Cross-validation is mainly an evaluation and model-selection tool. It helps reveal whether a choice generalizes across folds. The actual complexity control may come from regularization, pruning, early stopping, a simpler model, better features, more representative data, or another intervention.
Variance connection: overfitting is often associated with high variance. A high-variance model can change substantially when the training sample changes slightly, causing unstable predictions or fitted relationships.
Bias-Variance Tradeoff
Increasing model flexibility can reduce systematic error but can also make a model more sensitive to the particular training sample. The useful operating point is the complexity that gives the best generalization for the data and metric that matter—not a mathematically universal “perfect middle.”
| Feature | Underfitting | Useful Fit | Overfitting |
|---|---|---|---|
| Training performance | Poor | Good | Often very good |
| Validation performance | Poor | Good | Much worse than training |
| Bias tendency | Often high | Appropriate for task | Often lower |
| Variance tendency | Often lower | Controlled | Often high |
Learning Curves — Another Diagnostic
A learning curve compares training and validation performance as the amount of training data changes. It can help answer a practical question: Would more data likely help, or is the model itself too limited?
Typical underfitting pattern
Training and validation scores converge to similarly poor values.
Simply adding more data may not solve a model that cannot represent enough of the relationship.
Typical overfitting / high-variance pattern
Training score remains much better than validation score.
More representative data can sometimes narrow the gap, alongside complexity control.
Scikit-learn tools
learning_curve computes cross-validated training and validation scores for different training-set sizes. validation_curve computes training and validation scores while varying one model parameter. These are diagnostic tools; they do not automatically choose the final model for you.
Python Example — See Underfitting, Useful Fit, and Overfitting
The earlier clean equation y=x² is excellent for teaching curves, but it is not a good demonstration of real overfitting because it contains no random noise and no held-out evaluation. The example below creates a reproducible noisy relationship and compares three polynomial degrees on separate training and test data.
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.metrics import mean_squared_error
# 1. Reproducible noisy relationship
rng = np.random.RandomState(42)
X = np.linspace(0, 1, 40).reshape(-1, 1)
y = np.sin(2 * np.pi * X[:, 0]) + rng.normal(0, 0.15, 40)
# 2. Keep unseen data for evaluation
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.30,
random_state=42
)
# 3. Compare different model complexities
for degree in [1, 3, 15]:
model = make_pipeline(
PolynomialFeatures(degree=degree, include_bias=False),
LinearRegression()
)
model.fit(X_train, y_train)
train_mse = mean_squared_error(
y_train,
model.predict(X_train)
)
test_mse = mean_squared_error(
y_test,
model.predict(X_test)
)
print(
f"Degree {degree:>2}: "
f"train MSE = {train_mse:.3f}, "
f"test MSE = {test_mse:.3f}"
)Degree 1: train MSE = 0.170, test MSE = 0.290
Degree 3: train MSE = 0.018, test MSE = 0.022
Degree 15: train MSE = 0.009, test MSE = 1.187
Degree 1
Training and test error are both relatively high.
Diagnosis: underfitting in this example.
Degree 3
Training and test error are both low and close.
Diagnosis: useful generalization for this split.
Degree 15
Training error is lowest, but unseen-data error becomes much worse.
Diagnosis: overfitting in this example.
Do not memorize the degrees
Degree 1, 3, and 15 are only examples for this generated dataset. A degree-15 model is not automatically overfitted on every problem, and degree 3 is not universally best. Use validation or cross-validation to choose complexity for the actual data.
Match the Fix to the Diagnosis
| Observation | Possible next checks |
|---|---|
| Training and validation both poor | More suitable features, more expressive model, less excessive regularization, better optimization. |
| Training strong, validation much weaker | Regularization, simpler model, pruning, early stopping, feature selection, more representative data. |
| Both reasonably strong | Confirm across folds/test data and optimize only if the business/technical metric still requires improvement. |
Common Mistakes
“99% training accuracy means the model is excellent.”
Training performance alone cannot tell you whether the model generalizes.
“A complex model is always overfitted.”
Complexity increases risk, but overfitting is diagnosed from unseen-data performance.
“Cross-validation prevents overfitting.”
Cross-validation helps estimate and compare generalization; complexity control is a separate step.
“More data always fixes overfitting.”
More representative data can help some high-variance problems, but it is not a universal cure.
“The test set can be checked after every model change.”
Repeatedly adapting to test results turns the test set into part of model selection.
Quick Recap
Underfitting signal
Training and validation performance are both poor.
Overfitting signal
Training is much stronger than validation/test performance.
Main objective
Choose the model and complexity that generalize best to unseen data.
Final Summary
Underfitting and overfitting describe two important ways a model can fail to generalize. Underfitting means the model has not captured enough useful structure; overfitting means training-specific details influence the model too strongly. The most useful diagnosis comes from comparing training performance with properly held-out validation or cross-validation performance.
Most Important Insight to Remember
Do not choose a model because it fits the training data best. Choose it because its performance on unseen data shows that it has learned a useful, repeatable pattern.