Grid & Random Search

Grid Search and Random Search

A Machine Learning model can behave very differently when we change settings such as tree depth, number of neighbors, regularization strength, or learning rate. These configurable settings are called hyperparameters.

Parameters vs Hyperparameters

Hyperparameters influence model capacity, regularization, optimization, or computational cost. A useful value depends on the dataset, metric, model, and validation procedure—there is rarely one universal best setting.

What Is Grid Search?

Grid Search evaluates the Cartesian product of the candidate values you explicitly provide. If the grid contains 3 values for one hyperparameter and 2 for another, there are:

How Grid Search Works

Search-Cost Example

Suppose we search 3 values of C and 3 values of gamma:

This multiplication is why an exhaustive grid can become expensive when we add many hyperparameters or many candidate values.

What Is Random Search?

Random Search samples a fixed number of candidate settings from the search space rather than evaluating every combination. In Scikit-learn, n_iter controls how many parameter settings are tried.

Why Can Random Search Be Useful?

Not every hyperparameter influences performance equally. If one dimension matters a lot and another matters little, a rigid grid can spend many evaluations repeating the same few important values. Random sampling can explore more distinct values of important dimensions for the same evaluation budget.

from scipy.stats import loguniform

param_distributions = {
    "svc__C": loguniform(1e-2, 1e2),
    "svc__gamma": loguniform(1e-4, 1e0)
}

Grid Search vs Random Search

Why Cross-Validation Is Used During Search

Choosing hyperparameters from one validation split can make the result depend heavily on that particular split. Cross-validation rotates the validation fold and averages the scores, reducing dependence on one partition of the data.

These are validation folds, not the final test set. Keep the final test set outside the tuning loop whenever you want an independent estimate after model selection.

Avoid Preprocessing Leakage

If preprocessing learns from the full dataset before cross-validation, validation folds can influence quantities such as means, standard deviations, imputations, feature selection, or PCA components.

X_scaled = scaler.fit_transform(X)
GridSearchCV(model, ...).fit(X_scaled, y)

Pipeline([
    ("scale", StandardScaler()),
    ("svc", SVC())
])

The Pipeline ensures that preprocessing is fitted again inside each training fold. Pipeline parameters are addressed using names such as svc__C and svc__gamma.

Python: Grid and Random Search on the Same Model

This example uses Iris only as a small teaching dataset. We first hold out a final test set, then run both searches only on the training portion.

from sklearn.datasets import load_iris
from sklearn.model_selection import (
    train_test_split,
    GridSearchCV,
    RandomizedSearchCV,
    StratifiedKFold,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)

pipeline = Pipeline([
    ("scale", StandardScaler()),
    ("svc", SVC()),
])

cv = StratifiedKFold(
    n_splits=5,
    shuffle=True,
    random_state=42,
)

search_space = {
    "svc__C": [0.1, 1, 10],
    "svc__gamma": ["scale", 0.1, 0.01],
}

# Grid Search: all 3 x 3 = 9 combinations
grid = GridSearchCV(
    estimator=pipeline,
    param_grid=search_space,
    cv=cv,
    scoring="accuracy",
    refit=True,
    n_jobs=-1,
)
grid.fit(X_train, y_train)

# Random Search: sample only 4 combinations
random_search = RandomizedSearchCV(
    estimator=pipeline,
    param_distributions=search_space,
    n_iter=4,
    cv=cv,
    scoring="accuracy",
    random_state=42,
    refit=True,
    n_jobs=-1,
)
random_search.fit(X_train, y_train)

print("Grid candidates:", len(grid.cv_results_["params"]))
print("Grid best:", grid.best_params_)
print("Grid CV accuracy:", round(grid.best_score_, 3))
print("Grid test accuracy:", round(
    accuracy_score(y_test, grid.predict(X_test)), 3
))

print()
print("Random candidates:", len(random_search.cv_results_["params"]))
print("Random best:", random_search.best_params_)
print("Random CV accuracy:", round(random_search.best_score_, 3))
print("Random test accuracy:", round(
    accuracy_score(y_test, random_search.predict(X_test)), 3
))

Grid candidates: 9
Grid best: {'svc__C': 1, 'svc__gamma': 0.1}
Grid CV accuracy: 0.975
Grid test accuracy: 0.967

Random candidates: 4
Random best: {'svc__gamma': 0.1, 'svc__C': 10}
Random CV accuracy: 0.967
Random test accuracy: 0.967

Common Mistakes

Quick Recap

Final Summary

Grid Search exhaustively checks every combination in a defined finite grid. Random Search samples a fixed number of candidate settings, giving direct control over the search budget.

Neither method guarantees the universally best model. Good hyperparameter search also depends on a sensible search space, the right validation strategy, the right metric, leakage-safe preprocessing, and an untouched final evaluation when needed.