Decision Tree
Decision Trees are algorithms that work by repeatedly asking questions about the data and splitting it into smaller groups until a final prediction is made.
Decision Tree in Simple Words
Think of a Decision Tree as a sequence of simple questions. Each answer sends us to the next question until we reach a final decision.
Simple Example: Should We Play Cricket?
No ↓
Yes ↓
1. Ask
Ask one useful question about a feature.
2. Split
Send different answers into different groups.
3. Predict
Keep asking until the tree reaches a final prediction.
A Decision Tree looks similar to a flowchart:
- Each internal node represents a question
- Each branch represents an answer
- Each leaf node represents the final prediction
Decision Trees are widely used because their rules are easy to inspect, they can model nonlinear relationships, and the same tree idea can be used for both classification and regression.
What is a Decision Tree?
A Decision Tree is a supervised machine learning algorithm that divides data into smaller subsets based on conditions. The model learns a sequence of decision rules from historical data and uses those rules to make predictions on new data.
Simple Decision Tree Workflow
Real-Life Example: Bank Loan
Suppose a bank wants to decide whether to approve a loan. The tree learns rules automatically from previous customer records.
Structure of a Decision Tree
A Decision Tree mainly contains three types of nodes, forming a hierarchical structure.
1. Root Node
Topmost node containing the training samples before the first split. The algorithm searches possible feature-and-threshold splits and chooses one according to its splitting criterion.
Is Age > 30?
2. Decision Node
Internal nodes where additional conditions are evaluated to divide dataset into smaller, purer groups.
Is Income > 60K?
3. Leaf Node
Represents the final prediction for samples that reach that node. A leaf does not have to be perfectly pure; tree-stopping rules may end splitting earlier.
Loan Approved
Types of Decision Trees
Classification Tree
Predicts discrete categories or labels.
Examples
- YES / NO
- SPAM / NOT SPAM
- FRAUD / LEGITIMATE
Applications
- Spam detection
- Medical diagnosis
- Customer churn
Regression Tree
Predicts continuous numerical values.
Examples
- House Price
- Temperature
- Sales Forecast
Applications
- Real estate pricing
- Demand forecasting
- Revenue estimation
How Decision Trees Work & Purity
The main objective is to repeatedly divide the dataset into smaller and purer groups. At each node, the algorithm compares possible questions and chooses a split that improves the chosen impurity or prediction-loss criterion the most.
See the Goal Before the Formula
Before a useful split
YES · NO · YES · NO · YES · NO
Classes are mixed together.
YES · YES · YES
NO · NO · NO
Better split → purer child groups
Understanding Purity
Pure Node Example
All samples belong to the same category.
Impure Node Example
The node contains mixed categories.
Splitting Criteria (Math)
A classification tree needs a mathematical method to compare possible splits. Two common classification criteria are Gini impurity and Entropy / Information Gain.
1. Gini Index
Measures how mixed the classes are inside a classification node. Lower values mean greater purity. Scikit-learn uses Gini impurity as the default classification criterion. The formula is:
Where pᵢ = probability of class i
| Gini Value | Interpretation |
|---|---|
| 0 | Perfectly pure node |
| 0.5 | Most mixed binary node (50/50) |
For more than two classes, the maximum possible Gini impurity is higher than 0.5 and depends on the number of classes.
Worked Example of Gini Index
Suppose a node contains 6 YES and 4 NO (Total: 10)
P(NO) = 4/10 = 0.4
Gini = 1 - (0.36 + 0.16)
For a binary node, 0.48 is close to the maximum of 0.5, so the classes are still highly mixed.
2. Entropy & Information Gain
Entropy measures disorder or uncertainty inside a node (from Information Theory). Lower entropy means more organized data. Information Gain measures how much Entropy decreases after a split.
| Value | Meaning | Goal |
|---|---|---|
| 0 | Perfectly pure | Minimize |
| 1 | Max disorder (binary) | |
| High Gain | Huge uncertainty drop | Maximize |
Worked Example of Entropy
Suppose a node contains 6 YES and 4 NO (Total: 10)
log₂(0.6) ≈ -0.737, log₂(0.4) ≈ -1.322
0.4 * -1.322 = -0.5288
For two classes, entropy 0.971 is close to the maximum of 1, so the node is highly mixed.
Tiny Information Gain Example
Suppose a question splits the 10 samples into a pure child with 4 YES samples and another child with 2 YES + 4 NO samples.
Step 1: Parent entropy = 0.971
Step 2: Child entropies ≈ 0 and 0.918
Step 3: Weighted child entropy = (4/10 × 0) + (6/10 × 0.918) ≈ 0.551
Step 4: Information Gain = 0.971 − 0.551 ≈ 0.420
The question reduced uncertainty by about 0.420 in this small example.
Advanced: Overfitting & Limitations
Overfitting
Decision Trees can grow very deep and fit details that do not generalize well. This is overfitting: training performance may look strong while performance on new data gets worse.
- Creates very deep, complex branches
- Captures noise and outliers
- Leads to poor test accuracy
Pruning (One Control Strategy)
Tree complexity can be controlled while growing the tree or by pruning branches afterward. The goal is to keep useful structure without fitting unnecessary detail.
- Control growth: use settings such as max_depth, min_samples_split, or min_samples_leaf.
- Post-pruning: cost-complexity pruning can remove branches using a parameter such as ccp_alpha.
Advantages vs Disadvantages
| ✅ Advantages | ❌ Disadvantages |
|---|---|
| Interpretable: Human-readable rules | Overfitting: Deep trees can fit noise and overly specific patterns |
| Scaling usually unnecessary: splits rely on feature thresholds rather than distances | Instability: Small data changes can sometimes produce a noticeably different tree |
| Handles nonlinear relationships natively | Greedy splitting: Each split is chosen locally, so the final tree is not guaranteed to be globally optimal |
Python Implementation
Here is a practical Python implementation using Scikit-Learn.
Interactive: Train a Decision Tree in Python
Tune the maximum depth and splitting criterion to inspect how the tree structure and decision rules change in real-time.
Limits how many levels of conditional splits the tree can grow.
Mathematical metric used to evaluate the purity of candidate splits.
| 1 | import pandas as pd |
| 2 | from sklearn.model_selection import train_test_split |
| 3 | from sklearn.tree import DecisionTreeClassifier, export_text |
| 4 | from sklearn.metrics import accuracy_score |
| 5 | |
| 6 | # 1. Dataset |
| 7 | data = { |
| 8 | "Age": [22, 25, 28, 31, 35, 38, 42, 45, 48, 52, 55, 60], |
| 9 | "Income": [18000, 22000, 30000, 35000, 42000, 50000, 58000, 65000, 72000, 80000, 90000, 100000], |
| 10 | "Buy": [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1] |
| 11 | } |
| 12 | df = pd.DataFrame(data) |
| 13 | X = df[["Age", "Income"]] |
| 14 | y = df["Buy"] |
| 15 | |
| 16 | # 2. Train-Test Split |
| 17 | X_train, X_test, y_train, y_test = train_test_split( |
| 18 | X, y, test_size=0.33, random_state=42, stratify=y |
| 19 | ) |
| 20 | |
| 21 | # 3. Train Decision Tree |
| 22 | max_depth = 3 |
| 23 | criterion = 'gini' |
| 24 | model = DecisionTreeClassifier(max_depth=3 |
| 25 | model.fit(X_train, y_train) |
| 26 | |
| 27 | # 4. Predict and Inspect Learned Rules |
| 28 | predictions = model.predict(X_test) |
| 29 | acc = accuracy_score(y_test, predictions) |
| 30 | |
| 31 | print(f"Max Depth: {max_depth}") |
| 32 | print(f"Splitting Criterion: {criterion}") |
| 33 | print(f"Test Accuracy: {acc:.2f}") |
| 34 | print("\nLearned Decision Rules (ASCII Tree):") |
| 35 | print(export_text(model, feature_names=list(X.columns))) |
Expected Output
|--- Age <= 36.50
| |--- class: 0
|--- Age > 36.50
| |--- class: 1
This is a deliberately simple teaching dataset, so the test split is easy. A perfect score here does not mean Decision Trees will achieve 100% accuracy on real-world data.
Common Questions About Decision Trees
Do Decision Trees need feature scaling?
Usually no. A tree compares feature values with thresholds, so ordinary rescaling does not change the ordering of values. Categorical data may still need suitable preprocessing depending on the implementation.
What does max_depth do?
It limits how many levels the tree can grow. A smaller depth can make the model easier to interpret and can help control overfitting.
Decision Tree or Random Forest?
A single tree is easier to inspect. A Random Forest combines many trees and often gives more stable predictions, at the cost of some simplicity.
Final Summary
Decision Trees are an intuitive supervised-learning model because their predictions can be followed as a sequence of feature-based questions and branches.
A single tree can model nonlinear relationships and usually does not need ordinary feature scaling, but an unconstrained tree can overfit. Decision Trees are also important building blocks for ensemble methods such as Random Forest and Gradient Boosting.
Most Important Insight to Remember:
A Decision Tree is easy to inspect, but controlling its depth and validating it on unseen data are important because a single tree can be sensitive to the training sample.