Hyperparameter Tuning
A Machine Learning algorithm can behave very differently depending on the settings we choose for it. A Decision Tree can be shallow or deep. KNN can use 1 neighbor or 15 neighbors. A Neural Network can learn with a small or large learning rate. These user-controlled settings are called hyperparameters.
Hyperparameter Tuning in Simple Words
Choose Settings
Example: depth = 2, 4, 8
Train
Fit each candidate
Validate
Compare unseen-fold scores
Select
Keep the best candidate
Main idea: Hyperparameter tuning does not change the training data. It compares different model configurations and chooses the configuration that performs best according to a validation strategy and metric.
Real-Life Analogy: Preparing Tea
Milk, water and tea leaves are like the data. The recipe is like the algorithm. Settings such as boiling time and sugar quantity are like hyperparameters. The same ingredients can produce different results when those settings change. The goal is not to find one universally perfect recipe, but a recipe that works well for the situation we care about.
Too Simple
Example: Decision Tree max_depth=1 may underfit.
Useful Complexity
A validated setting may generalize better.
Too Flexible
A very deep tree may fit training-specific noise.
Important: Tuning can improve a model, but it cannot create useful signal that is absent from the data. A well-tuned weak representation can still perform poorly, and a larger search is not automatically a better search.
Parameters vs Hyperparameters
Model parameters are learned from data during fitting. Hyperparameters configure the learning procedure or model structure and are selected outside the model's ordinary parameter-fitting step—by a person, a search procedure, or another optimization process.
| Question | Parameters | Hyperparameters |
|---|---|---|
| How are they obtained? | Learned by fitting the model to data | Chosen/configured outside ordinary parameter fitting |
| Linear model example | Coefficients and intercept | Regularization strength |
| Decision Tree example | Chosen split features and thresholds | max_depth, min_samples_leaf |
| Neural Network example | Weights and biases | Learning rate, batch size, architecture choices |
A useful nuance
Textbooks often say hyperparameters are “set before training.” That is a useful beginner definition, but some training schedules can change values during training. The key distinction is that these settings are not learned in the same way as the model's fitted parameters.
Common Hyperparameter Categories
Model Complexity / Structure
- Decision Tree:
max_depth,min_samples_leaf - KNN:
n_neighbors - Random Forest:
max_depth,max_features - Neural Network: number and size of hidden layers
Training / Optimization
- Learning rate
- Batch size
- Maximum epochs / iterations
- Momentum or optimizer settings
Regularization
- Ridge/Lasso regularization strength
- SVM
C - Dropout rate
- Tree pruning / minimum-sample controls
Ensemble Settings
- Number of trees / boosting rounds
- Boosting learning rate
- Row or feature subsampling
- Base-estimator complexity
A Few Algorithm-Specific Examples
| Algorithm | Examples of hyperparameters |
|---|---|
| KNN | n_neighbors, distance metric, weights |
| Decision Tree | max_depth, min_samples_split, criterion |
| Random Forest | n_estimators, max_depth, max_features |
| SVM | C, kernel, gamma |
| Gradient Boosting | n_estimators, learning rate, tree depth |
How Does Tuning Choose a Setting?
Imagine we are tuning max_depth for a Decision Tree. We evaluate three candidate values using the same validation procedure.
| Candidate | Mean validation accuracy | Interpretation |
|---|---|---|
max_depth=2 | 0.84 | May be too simple |
max_depth=4 | 0.91 | Best of these candidates |
max_depth=8 | 0.87 | More complexity did not help validation |
Step-by-step
- Compare the candidate validation scores:
0.84, 0.91, 0.87. - The largest score is
0.91. - Among these candidates, select
max_depth=4. - Do not call it universally optimal. It is only the best setting among the candidates and validation procedure we tested.
Going Deeper: tuning as an optimization problem
Conceptually, for a score that we want to maximize:
Here, H is the search space and h* is the best candidate found by that search. If we are minimizing a loss such as RMSE, the direction is reversed.
Hyperparameter Tuning Workflow
A Safer 8-Step Workflow
- Keep a final test set aside: Do not use it to repeatedly choose hyperparameters.
- Choose a baseline model: Measure where you are starting.
- Choose the evaluation metric: Accuracy is not suitable for every task.
- Define a sensible search space: Include plausible values, not every imaginable number.
- Choose a search strategy: Manual, Grid Search, Random Search, or another suitable optimizer.
- Evaluate candidates using validation/CV: Use the same fair procedure for all candidates.
- Refit the selected configuration: Train the chosen configuration on the available development/training data.
- Evaluate once on the untouched test set: Report how the selected model performs on data not used for selection.
Do not tune on the final test set
If you try a configuration, check the test score, change the model, check the same test set again, and repeat, the test set gradually becomes part of the model-selection process. Use validation or cross-validation for tuning and reserve the final test set for final evaluation.
Major Tuning Techniques
Manual Search
Try a small number of settings based on prior knowledge and inspect validation results. Useful for learning and quick baselines, but inefficient for large search spaces.
Grid Search
Evaluate every combination in a predefined finite grid. Systematic, but the number of combinations can grow quickly.
Random Search
Evaluate a chosen number of sampled configurations. It can explore a larger space without evaluating every possible combination.
The Search-Budget Problem
Suppose our grid contains:
n_estimators: 2 valuesmax_depth: 3 valuesmin_samples_split: 2 values
With 5-fold cross-validation, that means 12 × 5 = 60 cross-validation fits, before considering the final refit of the selected configuration.
The next lesson, Grid & Random Search, compares these two search strategies in more detail.
GridSearchCV Example
This example keeps 20% of Iris as a final test set. GridSearchCV searches only inside the training portion using 5-fold stratified cross-validation.
Random Forest Hyperparameter Tuning
Pythonfrom sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import (
GridSearchCV,
StratifiedKFold,
train_test_split
)
# 1. Load data
X, y = load_iris(return_X_y=True)
# 2. Keep the final test set out of tuning
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.20,
random_state=42,
stratify=y
)
# 3. Baseline estimator
model = RandomForestClassifier(random_state=42)
# 4. Candidate hyperparameters: 2 x 3 x 2 = 12 combinations
param_grid = {
"n_estimators": [50, 100],
"max_depth": [2, 4, None],
"min_samples_split": [2, 5]
}
# 5. Validation strategy
cv = StratifiedKFold(
n_splits=5,
shuffle=True,
random_state=42
)
# 6. Search on training data only
search = GridSearchCV(
estimator=model,
param_grid=param_grid,
scoring="accuracy",
cv=cv,
n_jobs=-1,
refit=True
)
search.fit(X_train, y_train)
# 7. Evaluate selected model on untouched test data
pred = search.predict(X_test)
print("Best parameters:", search.best_params_)
print("Best CV accuracy:", round(search.best_score_, 3))
print("Final test accuracy:", round(accuracy_score(y_test, pred), 3))Output:
Best parameters: {'max_depth': 4, 'min_samples_split': 5, 'n_estimators': 50}
Best CV accuracy: 0.967
Final test accuracy: 0.933
How to read this result
- Best parameters are the best combination among the 12 candidates under this CV procedure.
- Best CV accuracy = 0.967 is a model-selection score from the training portion.
- Final test accuracy = 0.933 is a separate evaluation on data that was not used by GridSearchCV.
- The lower test score is not automatically a problem; validation and test scores naturally differ because they are based on different samples.
What does refit=True do?
After the search identifies the best hyperparameters, GridSearchCV refits that configuration using the full data supplied to search.fit(...). In this example that is the training portion—not the held-out test set.
Tuning with Preprocessing: Avoid Leakage
If a model needs learned preprocessing such as scaling, imputation, PCA, or feature selection, that preprocessing should be fitted inside each training fold—not once on the complete dataset before cross-validation.
Wrong pattern
X_scaled = scaler.fit_transform(X) GridSearchCV(model, ...).fit(X_scaled, y)
The scaler has already learned statistics from rows that later appear in validation folds.
Safer pattern
Pipeline([
("scale", StandardScaler()),
("model", SVC())
])The pipeline fits preprocessing separately inside each training fold.
When tuning a parameter inside a Pipeline, Scikit-learn uses names such as model__C: the step name, two underscores, then the estimator parameter name.
Tune for the Right Metric
A search procedure selects the configuration that looks best according to the metric you give it. Therefore the metric should reflect the problem you actually care about.
Balanced classification
Accuracy may be reasonable.
Rare positive cases
Precision, recall, F1, PR-style metrics, or a business cost may matter more.
Regression
MAE, RMSE, R², or domain-specific error may be appropriate.
The best model depends on the metric
Configuration A can have better accuracy while Configuration B has better recall. Hyperparameter tuning cannot decide which business trade-off matters—you must define that goal first.
Common Mistakes
1. Tuning on the test set
Use validation/CV for selection; keep final test data out of repeated decisions.
2. Searching an enormous grid without a reason
More combinations mean more compute and can increase model-selection overfitting. Start with sensible ranges.
3. Ignoring preprocessing leakage
Put learned preprocessing inside a Pipeline when it is part of the model-selection workflow.
4. Assuming the best CV score is the final real-world score
It is an estimate used for selection. Confirm the selected procedure on untouched data.
5. Tuning everything at once
A smaller, meaningful search space is often easier to interpret and cheaper to validate.
Going Deeper: Selection Bias and Nested CV
If we evaluate many configurations and report the very best cross-validation score from the same search as though it were an unbiased final performance estimate, that score can be optimistic because the configuration was selected to look good on those folds.
For higher-stakes comparisons, nested cross-validation can place hyperparameter selection in an inner loop and performance estimation in an outer loop. For ordinary projects, a clean train/validation-or-CV/test workflow is often easier to understand and implement.
Quick Recap
Is a hyperparameter learned like a model coefficient?
No. Hyperparameters configure the model/training process and are selected outside ordinary parameter fitting.
Why should the final test set stay outside GridSearchCV?
Because using it to choose settings would make the test set part of model selection and weaken its role as an unseen final evaluation.
Does a larger search always produce a better model?
No. Search quality depends on the candidate space, metric, validation design, data, model family, and compute budget.
Final Summary
Hyperparameters control choices such as model complexity, regularization, neighborhood size, learning rate and ensemble behavior. Hyperparameter tuning compares candidate configurations using a validation strategy so that we can select a configuration that is more likely to generalize well.
The essential workflow is: define the metric → define a sensible search space → tune using validation/cross-validation → refit the selected configuration → evaluate on untouched test data. Grid Search is one search method; Random Search and other optimizers can be more practical when the search space becomes large.
Most Important Insight to Remember
Hyperparameter tuning is not about finding a magical universally best setting. It is about making a fair, validation-based choice among candidate configurations for a specific dataset, metric and modeling goal.