Bagging (Bootstrap Aggregating)
Bagging means Bootstrap Aggregating: train several versions of a model on different random samples of the training data, then combine their predictions.
Bagging in Simple Words
A single model—especially an unstable model such as a deep Decision Tree—can change noticeably when the training data changes a little. Bagging tries to make the final prediction more stable by letting several models learn from slightly different versions of the training set.
Original Data
Start with one training dataset.
Bootstrap Samples
Create new samples by drawing rows with replacement.
Train Models
Fit one base model on each sample.
Combine
Average or vote to make the final prediction.
Tiny Example: Why Different Samples?
| Dataset | Rows | What to notice |
|---|---|---|
| Original | [A, B, C, D, E] | Five training rows |
| Sample 1 | [B, B, D, E, A] | B repeats; C is absent |
| Sample 2 | [C, A, A, D, B] | A repeats; E is absent |
Each model sees a slightly different training sample, so the models do not all make exactly the same mistakes.
The key idea
Bagging is most useful when the base model has high variance. Combining diverse predictions can reduce that variance, although it does not guarantee better performance on every dataset.
Simple Real-Life Analogy: A Committee
Suppose a company asks only one expert for an important decision. That expert may miss something. If several experts study slightly different evidence and the company combines their opinions, one person's unusual mistake has less influence on the final result.
Bagging follows a similar idea: create diversity among models and aggregate their predictions. The benefit depends on how accurate and how different the individual models are.
Why Bagging Is Useful: Reducing Variance
Bagging is especially useful for models with high variance. Such a model may fit one training sample very well but change substantially when the sample changes. This can contribute to overfitting and weaker generalization.
A simple analogy is a student who memorizes previous answers instead of learning the underlying ideas. Performance may look excellent on familiar questions but drop when the questions change. A high-variance model can show a similar gap between training behaviour and unseen data.
High Variance Models
Some algorithms are more sensitive to the exact training sample than others. A small data change can produce a noticeably different fitted model. This sensitivity is associated with high variance. Fully grown Decision Trees are a classic example, which is why they are often used to demonstrate Bagging.
Main Goal of Bagging
The main objective of Bagging is to reduce the variance of an unstable base estimator by aggregating several randomized versions of it.
Averaging can make predictions more stable when the individual models make partly different errors. The bias–variance trade-off still depends on the estimator and the data.
Understanding Bootstrap Aggregating
The word Bagging contains two major concepts: Bootstrap and Aggregating. Both are extremely important.
High-Level Workflow of Bagging
- Original Dataset
- Create Multiple Bootstrap Samples: Draw training rows with replacement, so rows can repeat.
- Train Multiple Base Models: Often Decision Trees.
- Each Model Makes Prediction
- Aggregate Predictions: Using majority voting or averaging.
- Final Prediction
What is Bootstrap?
Bootstrap means random sampling with replacement. A row can therefore be drawn more than once, while another row may not appear in a particular bootstrap sample.
Example:
Original dataset: [1, 2, 3, 4, 5]
Bootstrap sample 1: [2, 2, 4, 5, 1]
Bootstrap sample 2: [3, 1, 1, 4, 2]
Bootstrap sample 3: [5, 3, 2, 2, 4]
Notice: some rows repeat and some are left out. That is expected when sampling with replacement.
Going Deeper: Why do some rows get left out?
If a bootstrap sample makes N draws from N rows, the probability that one particular row is never selected is:
(1 - 1/N)N ≈ e-1 ≈ 0.368
So for a large dataset, roughly 36.8% of rows are left out of a particular bootstrap sample and roughly 63.2% appear at least once. Those left-out rows are the basis of out-of-bag evaluation.
What is Aggregating?
After training multiple models, their predictions must be combined. This is Aggregation.
- Classification: A simple mental model is majority voting. In Scikit-learn's
BaggingClassifier, if the base estimators provide class probabilities, the class with the highest mean predicted probability is selected; otherwise it falls back to voting. - Regression: The base regressors' numerical predictions are averaged to form the ensemble prediction.
Visualizing Aggregation
Classification: Simple Voting Intuition
Five model predictions for the same new example
Yes = 3 votes | No = 2 votes
Simple majority result → Yes
This is the easiest intuition. Probability-aware classifiers may aggregate mean class probabilities instead.
Regression: Averaging Example
Predictions: 100, 120, 110, 130, 140
100 + 120 + 110 + 130 + 140 = 600
600 / 5 = 120
Final prediction = 120
Going Deeper: Why averaging can reduce variance
If n independent models each have variance σ², the variance of their average is:
Var(average) = σ² / n
Example: if σ² = 100 and n = 5, the ideal independent-model variance becomes 100 / 5 = 20.
Real bagged models are usually correlated because they come from the same underlying dataset. With equal pairwise correlation ρ, a useful approximation is σ²[ρ + (1-ρ)/n]. So adding models can reduce variance, but it does not make variance disappear completely.
Bagging vs Random Forest vs Boosting
| Method | How models differ | How they are trained | Main intuition |
|---|---|---|---|
| Bagging | Different random samples; base estimator is configurable | Can be trained independently | Reduce variance through aggregation |
| Random Forest | Bagged trees plus extra randomness in feature selection at splits | Trees can be trained independently | Decorrelate trees further |
| Boosting | Later learners respond to earlier errors/residuals | Sequential | Build a strong model stage by stage |
Useful connections
Review Decision Trees and Bias–Variance, compare with Random Forest, then continue to Boosting.
Complete Bagging Pipeline
Here is the high-level progression from data collection to final output:
Python Implementation
This example compares one Decision Tree with a Bagging ensemble on a reproducible synthetic classification dataset. The train/test split is kept separate, and oob_score=True also gives an out-of-bag estimate from the training data.
Python Code: Decision Tree vs Bagging
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import BaggingClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
# Reproducible toy classification data
X, y = make_classification(
n_samples=500,
n_features=6,
n_informative=4,
n_redundant=1,
class_sep=1.0,
flip_y=0.05,
random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
X, y,
test_size=0.25,
random_state=42,
stratify=y
)
# One tree
single_tree = DecisionTreeClassifier(random_state=42)
single_tree.fit(X_train, y_train)
# Many trees trained on bootstrap samples
bag_model = BaggingClassifier(
estimator=DecisionTreeClassifier(random_state=42),
n_estimators=50,
bootstrap=True,
oob_score=True,
random_state=42,
n_jobs=-1
)
bag_model.fit(X_train, y_train)
single_pred = single_tree.predict(X_test)
bag_pred = bag_model.predict(X_test)
print(f"Single tree accuracy: {accuracy_score(y_test, single_pred):.3f}")
print(f"Bagging accuracy: {accuracy_score(y_test, bag_pred):.3f}")
print(f"OOB score: {bag_model.oob_score_:.3f}")Single tree accuracy: 0.680
Bagging accuracy: 0.824
OOB score: 0.805
Do not generalize from one toy run
Bagging performs better than the single tree in this particular reproducible example. It is not a rule that Bagging must improve every dataset or every base estimator. Compare models using appropriate validation on your own problem.
Important Scikit-learn Parameters
| Parameter | Simple meaning |
|---|---|
| estimator | The base model copied and fitted many times. |
| n_estimators | How many base models are in the ensemble. |
| max_samples | How many training samples each base model receives. |
| bootstrap=True | Draw samples with replacement—the classic Bagging setup. |
| max_features | Optionally restrict how many features each base model sees. |
| oob_score=True | Estimate generalization using samples left out of bootstrap draws; available when bootstrapping is used. |
When Is Bagging a Good Choice?
Bagging may help when...
- a single model changes a lot when the training sample changes,
- a flexible model such as a deep Decision Tree is overfitting,
- you can afford to train several base estimators, and
- validation shows that aggregation improves generalization.
Bagging may help less when...
- the base model already underfits badly,
- all base models make almost the same errors,
- the base estimator is extremely expensive to train many times, or
- the dataset lacks useful predictive signal.
Common Mistakes
Thinking bootstrap means sampling without replacement
Classic Bagging uses sampling with replacement, so duplicate rows are normal.
Assuming more estimators always keep giving large gains
Performance often levels off. More models also cost more time and memory.
Using the final test set to choose n_estimators or other settings
Tune with validation or cross-validation; keep the final test set for the final evaluation.
Confusing Bagging with Random Forest
Random Forest is a tree-specific ensemble that adds feature randomness to the bagging idea.
Treating OOB score as magic
OOB is a useful training-data estimate when bootstrapping is enabled, but it does not remove the need for careful evaluation.
Quick Recap
1. Why can the same row appear twice in a bootstrap sample?
Because bootstrap sampling draws rows with replacement.
2. What is Bagging mainly trying to reduce?
It is mainly used to reduce the variance of an unstable base estimator.
3. How are regression predictions combined?
The base regressors' numerical predictions are typically averaged.
Final Summary
Bagging is a general ensemble strategy that trains multiple versions of a base estimator on randomized samples and combines their predictions.
Its main strength is variance reduction: when the base estimators make partly different errors, aggregation can produce a more stable predictor than one fitted model alone.
Most Important Insight to Remember:
"Bagging = bootstrap samples + multiple base models + aggregation. It is especially useful when the base estimator is accurate enough but sensitive to changes in the training sample."