Cross-Validation
Cross-validation evaluates a model on several different held-out parts of the available training data. Instead of trusting one lucky or unlucky split, we rotate which observations are used for validation and summarize the results.
Main idea
Split into folds → train on most folds → validate on one fold → rotate → average the scores.
Cross-Validation in Simple Words
Split
Divide the development data into several folds.
Train
Train the model using all but one fold.
Validate
Score it on the fold that was left out.
Rotate
Repeat so every fold gets a turn.
Tiny 5-fold picture
T = training fold in that round. V = validation fold. Each sample is used for validation once in ordinary K-Fold.
Why Do We Need It?
Imagine judging a student's ability using only one short practice test. A very easy or unusually difficult test can give a misleading impression. Several different tests provide a broader picture. A single train/test split has the same weakness: the measured score can depend on which rows happened to land in the hold-out set.
Cross-validation does not automatically prevent overfitting. It gives repeated evidence about generalization and is useful for model comparison and hyperparameter selection. A model can still be over-tuned to repeated cross-validation decisions, which is why an untouched final test set can remain valuable.
Why One Split Can Be Misleading
Suppose three reasonable train/validation splits of the same dataset produce these illustrative accuracy values:
| Split | Accuracy |
|---|---|
| Split 1 | 92% |
| Split 2 | 84% |
| Split 3 | 88% |
Cross-validation reduces our dependence on one particular partition by evaluating several partitions. It does not remove all uncertainty, but it lets us see both the average score and how much the score changes across folds.
Worked average
Fold scores = 91%, 89%, 92%, 88%, 90%
Step 1: 91 + 89 + 92 + 88 + 90 = 450
Step 2: 450 ÷ 5 = 90
Mean validation accuracy = 90%
Cross-Validation and Validation Strategies
1. Hold-Out Validation
Hold-out validation makes one split into training and validation/test portions. It is fast and simple, but its estimate depends more strongly on that one split.
Ratios such as 80/20 or 70/30 are examples, not universal rules. Dataset size and the reliability needed from evaluation should guide the choice.
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.20,
random_state=42,
stratify=y
)2. K-Fold Cross-Validation
K-Fold divides the data into K folds. In each round, one fold is held out for validation and the other K − 1 folds are used for training. Every fold receives one validation turn.
Example: 100 samples, K = 5
Each fold contains about 20 samples.
- Each round trains on about 80 samples.
- Each round validates on about 20 samples.
- The process produces 5 validation scores.
Useful because
- Uses every sample for both training and validation across different rounds.
- Reduces dependence on one hold-out partition.
Trade-off
- Requires fitting the model K times.
- Fold choice must respect the structure of the data.
3. Stratified K-Fold
For binary or multiclass classification, Stratified K-Fold tries to preserve class proportions in each fold. This is particularly useful when one class is uncommon.
Stratification preserves proportions as closely as possible; exact percentages are not always possible because sample counts must be whole numbers. It also does not solve class imbalance by itself—it only makes the evaluation folds more representative of the label proportions.
from sklearn.model_selection import StratifiedKFold
cv = StratifiedKFold(
n_splits=5,
shuffle=True,
random_state=42
)4. Leave-One-Out Cross-Validation (LOOCV)
LOOCV sets K equal to the number of samples. With 10 observations, it fits 10 models: each round validates on one observation and trains on the other nine.
LOOCV uses nearly all observations for training in each round, but can be computationally expensive and is not automatically the best choice just because a dataset is small.
5. Leave-P-Out
Leave-P-Out validates on every possible set of P observations. The number of splits grows combinatorially.
Example: N = 5, P = 2
Number of validation pairs:
C(5, 2) = 10
For N = 100 and P = 2, this already becomes 4,950 splits.
6. Repeated K-Fold
Repeated K-Fold builds several randomized K-Fold partitions. For example, 5 folds repeated 10 times produce 50 validation scores. This can show how sensitive performance is to different randomized fold assignments, at additional computational cost.
7. Repeated Random Splits (Shuffle / Monte Carlo Style)
Instead of partitioning the data into one fixed set of non-overlapping folds, repeated random splitting creates fresh train/validation partitions multiple times. Across repetitions, a sample may appear in validation more than once or not at all.
8. Group-Aware Cross-Validation
If several rows belong to the same person, customer, machine, household, patient, or other entity, ordinary K-Fold can leak entity-specific information across train and validation folds. Group-aware splitting keeps a group out of both sides of the same split.
Example
If each patient has five measurements, all measurements from Patient 17 should generally stay together in either training or validation for a given split—not be divided between both.
from sklearn.model_selection import GroupKFold, cross_val_score
cv = GroupKFold(n_splits=5)
scores = cross_val_score(model, X, y, cv=cv, groups=patient_id)9. Time-Series Cross-Validation
Time-ordered data usually needs chronological splitting. Training on the future and validating on the past can create unrealistic leakage.
This expanding-window picture matches the basic idea behind TimeSeriesSplit: later training sets contain earlier training sets plus more historical data.
10. Nested Cross-Validation
Nested CV separates hyperparameter/model selection from performance estimation when you want a less optimistically biased estimate of a tuning procedure.
- Outer split: hold out one fold for evaluation.
- Inner CV: tune choices using only the outer-training portion.
- Refit: train the selected setup on that outer-training portion.
- Outer score: evaluate once on the outer held-out fold.
- Repeat across outer folds and summarize the outer scores.
Cross-Validation and Data Leakage
Cross-validation only gives a trustworthy estimate when every learned preprocessing step is fitted inside each training fold. Scaling, imputation, feature selection and PCA can all leak information if they are fitted once on the entire dataset before CV.
Wrong
X_scaled = scaler.fit_transform(X) # CV now receives data whose scaling # already learned from every row. cross_val_score(model, X_scaled, y, cv=cv)
Better
pipeline = make_pipeline(
StandardScaler(),
LogisticRegression(max_iter=1000)
)
cross_val_score(pipeline, X, y, cv=cv)A Scikit-learn Pipeline makes the scaler fit only on the training rows of each fold and then applies that fitted transformation to the fold being validated.
Complete Python Example
The example below uses Iris classification, a Pipeline, and an explicit shuffled Stratified K-Fold so the splitting behavior is visible rather than hidden behind cv=5.
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_iris(return_X_y=True)
cv = StratifiedKFold(
n_splits=5,
shuffle=True,
random_state=42
)
model = make_pipeline(
StandardScaler(),
LogisticRegression(max_iter=1000)
)
scores = cross_val_score(
model,
X,
y,
cv=cv,
scoring='accuracy'
)
print('Fold scores:', scores.round(3))
print('Mean accuracy:', round(scores.mean(), 3))
print('Std deviation:', round(scores.std(), 3))Expected output with these exact settings
Fold scores: [1. 0.967 0.9 1. 0.9 ] Mean accuracy: 0.953 Std deviation: 0.045
The mean says the five validation accuracies average about 95.3% for this educational example. The standard deviation gives a compact view of how much the five scores vary. These numbers are not a promised Logistic Regression accuracy on other datasets.
Choosing a Validation Strategy
| Situation | Useful starting strategy | Why |
|---|---|---|
| Ordinary classification | Stratified K-Fold | Preserves class proportions across folds. |
| Ordinary regression | K-Fold | Rotates held-out folds across the dataset. |
| Repeated rows per person/entity | Group-aware CV | Prevents the same group appearing on both sides of a split. |
| Time-ordered forecasting | Time-series CV | Respects chronology. |
| Evaluating a tuning procedure | Nested CV | Separates inner tuning from outer performance estimation. |
Cross-Validation vs One Train/Test Split
| Question | Single split | Cross-validation |
|---|---|---|
| How many evaluation partitions? | One | Several |
| Dependence on one split? | Higher | Reduced |
| Computation? | Lower | Higher because models are refit |
| Useful for model selection? | Possible, but sensitive to one validation split | Often more informative |
Common Mistakes
- Leakage before CV: fitting a scaler, imputer, PCA, or feature selector on all rows first.
- Ignoring groups: letting the same patient/customer/entity appear in training and validation.
- Random CV for time series: allowing future data to influence models evaluated on the past.
- Using accuracy blindly: an imbalanced classification problem may require precision, recall, F1, ROC-AUC, PR-AUC, or a business-specific metric instead.
- Thinking more folds are always better: more folds increase computation and change the statistical properties of the estimate.
- Tuning forever on the same CV: repeated decision-making can overfit the validation process itself.
A Useful Scikit-learn Detail
When cross_val_score(..., cv=5) receives an ordinary binary or multiclass classifier, Scikit-learn uses a stratified K-Fold strategy. For other cases it uses K-Fold. With an integer cv, those default splitters do not shuffle.
If you want shuffled folds with a fixed seed, create the splitter explicitly—as the Python example above does—and pass cv=cv. This also avoids a common coding mistake: creating a KFold object but then accidentally calling cross_val_score(..., cv=5), which ignores that object.
Quick Recap
Why not trust one train/test split?
Because the measured score can depend strongly on which rows happened to be held out. Cross-validation evaluates several held-out partitions.
Does cross-validation prevent overfitting?
No. It helps evaluate and select models more carefully, but models and even the model-selection process can still overfit.
Which splitter should I use for repeated measurements from the same patient?
Use a group-aware strategy so measurements from one patient do not leak across training and validation in the same split.
Final Summary
Cross-validation repeatedly trains and validates a model across different subsets of the available development data. Its main benefit is reducing dependence on one arbitrary split and giving a distribution of validation scores rather than one number.
The most important skill is not memorizing “K = 5.” It is choosing a split strategy that matches the data: stratify classification labels when appropriate, respect groups, preserve time order, keep learned preprocessing inside each fold, and retain a genuinely untouched final test set when you need a final performance estimate.