Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is a linear dimensionality-reduction technique. It transforms many original features into a smaller set of new features called principal components.
PCA in Simple Words
Start with many features
Find directions with large variation
Create new principal components
Keep only the useful components
Example: 4 correlated measurements → 2 principal components that retain most of the dataset's variance.
Important meaning of “information” in PCA
PCA preserves variance, not necessarily the information that is most useful for a prediction target. PCA is unsupervised: it does not look at class labels or target values while finding components.
Why Use PCA?
Modern datasets can contain many features, including features that carry overlapping information. Working with many dimensions can make analysis, visualization, storage, and some models more difficult.
Many features
A dataset may have tens, hundreds, or thousands of columns.
Correlated features
Several columns may describe similar underlying variation.
Hard to visualize
Humans cannot directly view 20-dimensional feature spaces.
Computational cost
Fewer dimensions can reduce later storage or computation in suitable workflows.
PCA and High-Dimensional Data
High dimensionality can create practical and statistical challenges, but PCA is only one possible tool for dimensionality reduction. It is not an automatic cure for every high-dimensional problem and does not guarantee better model accuracy.
Core Idea Behind PCA
PCA looks for new directions in the feature space. The first principal component (PC1) is the direction along which the projected data has the greatest variance. The next component captures the greatest remaining variance while being perpendicular (orthogonal) to earlier components.
Simple Visual: Rotate the Axes
PC1 follows the largest spread of the data. PC2 is perpendicular and captures the next-largest remaining variation.
Real-Life Analogy
Imagine a classroom dataset containing height, arm length, leg length, and shoe size. These measurements may be correlated because they partly reflect an underlying body-size pattern.
PCA may combine those correlated measurements into a component that we might informally interpret as “overall size.” But the component itself is mathematically a weighted combination of the original features; PCA does not automatically assign a human meaning to it.
PC1
Maximum projected variance.
PC2
Maximum remaining variance, orthogonal to PC1.
Later PCs
Continue in descending order of explained variance.
Important PCA Terminology
Feature
An original input column.
Variance
How spread out values are.
Covariance
How two variables vary together.
Eigenvector
A direction associated with the covariance structure.
Eigenvalue
Variance associated with that eigenvector.
Explained variance ratio
Fraction of total variance represented by a component.
Understanding Variance and Covariance
Variance
Variance measures how spread out values are around their mean. PCA orders directions by variance, but high variance does not automatically mean high predictive importance for a target variable.
Sample Variance
s² = [1 / (n - 1)] × Σ(xᵢ - x̄)²
For a sample, the common denominator is n - 1. Population variance instead uses n.
Worked Variance Example
Sample: 2, 4, 6
Mean = (2 + 4 + 6) / 3 = 4
Squared deviations = 4, 0, 4
Sum = 8
Sample variance = 8 / (3 - 1) = 4
Covariance
Covariance measures whether two variables tend to move together. Positive covariance means they tend to move in the same direction; negative covariance means they tend to move in opposite directions.
Positive Covariance
Example: in a dataset, height and leg length may tend to increase together.
Negative Covariance
Example: for a fixed distance, higher speed may be associated with lower travel time.
Sample Covariance
Cov(X,Y) = [1 / (n - 1)] × Σ(xᵢ - x̄)(yᵢ - ȳ)
Worked example
X = [2, 4, 6], x̄ = 4
Y = [60, 80, 100], ȳ = 80
Products = (-2)(-20), (0)(0), (2)(20)
Products = 40, 0, 40
Cov(X,Y) = 80 / 2 = 40
Covariance depends on scale
A feature measured in thousands can dominate a feature measured in small units. This is why standardization is often considered before PCA when feature scales are not naturally comparable.
PCA Step-by-Step Workflow
Suppose three students have Math and Physics scores below. Physics is exactly twice Math, so both features contain the same one-dimensional pattern.
| Student | Math (X) | Physics (Y) |
|---|---|---|
| A | 2 | 4 |
| B | 4 | 8 |
| C | 6 | 12 |
For this hand calculation we center the two features but do not standardize them. That keeps the arithmetic transparent and shows PCA on the covariance matrix.
Step 1 — Center the Data
Mean(Math) = (2 + 4 + 6) / 3 = 4
Mean(Physics) = (4 + 8 + 12) / 3 = 8
Centered points
- A: (2 - 4, 4 - 8) = (-2, -4)
- B: (4 - 4, 8 - 8) = (0, 0)
- C: (6 - 4, 12 - 8) = (2, 4)
Step 2 — Build the Covariance Matrix
Var(Math) = (4 + 0 + 4) / 2 = 4
Var(Physics) = (16 + 0 + 16) / 2 = 16
Cov(Math, Physics) = (8 + 0 + 8) / 2 = 8
Step 3 — Find Eigenvalues and Eigenvectors
Eigenvectors give candidate directions; eigenvalues tell us how much variance lies along those directions.
Characteristic equation
det(A - λI) = 0
(4 - λ)(16 - λ) - 64 = 0
λ² - 20λ = 0
λ(λ - 20) = 0
Therefore:
- λ₁ = 20
- λ₂ = 0
A direction for λ₁ is [1, 2]. PCA normally represents a component as a unit vector, so normalize it:
Length = √(1² + 2²) = √5
PC1 = [1/√5, 2/√5] ≈ [0.447, 0.894]
The opposite sign [-0.447, -0.894] represents the same axis; eigenvector signs are arbitrary.
Step 4 — Rank Components by Explained Variance
Total variance is `20 + 0 = 20`, so PC1 explains `20 / 20 = 100%` of the variance in this intentionally perfect-line example. PC2 explains 0%.
Step 5 — Project the Data onto PC1
Projection = centered point · unit PC1 vector
- A: (-2 × 0.447) + (-4 × 0.894) ≈ -4.472
- B: (0 × 0.447) + (0 × 0.894) = 0
- C: (2 × 0.447) + (4 × 0.894) ≈ 4.472
The 2D points are therefore represented in one dimension as approximately [-4.472, 0, 4.472] with no variance lost in this special toy example.
Should We Scale Before PCA?
PCA is sensitive to feature scale because it follows variance. If one feature is measured in thousands and another in decimals, the larger-scale feature can dominate the principal components.
Often standardize first when...
Features use different units or their raw scales should not determine importance.
You may keep original scale when...
The original variance scale itself is meaningful and intentionally should influence PCA.
Scikit-learn detail
`sklearn.decomposition.PCA` centers the features automatically, but does not standardize their scales. `StandardScaler` is a separate preprocessing choice.
Choosing How Many Components to Keep
A scree plot displays how much variance each principal component explains. It can help us inspect where additional components begin contributing relatively little variance.
Illustrative Explained Variance
Elbow idea
A bend can suggest a reasonable stopping point, but it is a heuristic—not a guaranteed “optimal” number.
Cumulative variance
In this illustration, PC1 + PC2 explain 85% + 10% = 95% of the total variance.
Real-Life Examples
Image Representation
Images can contain thousands of pixel features. PCA has historically been used to represent image variation with fewer components; the classic “eigenfaces” idea is one example. Reduced components can support visualization, compression, or downstream modeling.
Financial Data Exploration
Many financial variables may move together. PCA can summarize correlated patterns into fewer components for exploration or risk modeling, but those components should not automatically be interpreted as causal market factors.
Python Implementation of PCA
PCA on the Iris dataset
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
# 1. Load data
iris = load_iris()
X = iris.data
y = iris.target
# 2. Standardize deliberately before PCA
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
# 3. Reduce 4 features to 2 principal components
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
# 4. Inspect explained variance
print("Explained variance ratio:", pca.explained_variance_ratio_.round(4))
print("Cumulative explained variance:", round(pca.explained_variance_ratio_.sum(), 4))
# 5. Visualize the 2D projection
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y)
plt.xlabel("PC1")
plt.ylabel("PC2")
plt.title("PCA Projection of Iris Data")
plt.show()Verified output:
Explained variance ratio: [0.7296 0.2285] Cumulative explained variance: 0.9581
In this standardized Iris example, the first two components represent about 95.81% of total feature variance. That does not mean they preserve 95.81% of every possible task-relevant signal.
Avoid preprocessing leakage
The code above is an exploratory visualization of the full Iris dataset. If PCA is part of a predictive train/test workflow, fit both scaling and PCA on the training data only, ideally inside a Scikit-learn `Pipeline`.
Keep Enough Components for a Variance Target
# With the full SVD solver, keep enough components
# to explain more than 95% of the variance.
pca_95 = PCA(n_components=0.95, svd_solver="full")
X_reduced = pca_95.fit_transform(X_scaled)
print(pca_95.n_components_)Advantages & Disadvantages
Potential Advantages
- Can reduce the number of dimensions.
- Can summarize correlated feature variation.
- Can make high-dimensional data easier to visualize.
- May reduce downstream computation when many components are discarded.
Limitations
- Principal components can be harder to interpret than original features.
- Discarded low-variance directions can still contain predictive signal.
- PCA is a linear transformation and may not capture nonlinear structure well.
- Results can change substantially depending on feature scaling.
PCA does not guarantee less overfitting
Reducing dimensions can sometimes improve generalization, but it can also remove useful signal. For supervised tasks, choose the PCA configuration using a proper validation or cross-validation workflow rather than assuming that more compression is always better.
PCA vs Feature Selection
| Question | PCA | Feature Selection |
|---|---|---|
| What happens to features? | Creates new linear combinations | Keeps a subset of original features |
| Interpretability | Often lower | Often easier because original columns remain |
| Uses target labels? | No | Depends on the selection method |
Applications of PCA
Common PCA Mistakes
Assuming PCA automatically scales features
Scikit-learn PCA centers features, but scaling is a separate preprocessing decision.
Calling the highest-variance component the most predictive
PCA does not use the target and can discard a low-variance but predictive direction.
Using an elbow as an exact optimum
A scree-plot elbow is only one heuristic for choosing dimensionality.
Fitting PCA on the full dataset before supervised evaluation
In predictive workflows, fit preprocessing on training folds/data only to avoid leakage.
Quick Questions
Does PCA always require feature scaling?
No. But scaling is often appropriate when feature units or magnitudes differ and those raw scales should not determine PCA's variance directions.
Are principal components original features?
No. Each component is a new linear combination of the original features.
What should I learn after PCA?
Continue to t-SNE to see a nonlinear visualization method, and compare PCA with Feature Selection. Review Feature Scaling if the preprocessing choice is still unclear.
Final Summary
PCA is a linear dimensionality-reduction technique that creates orthogonal principal components and orders them by the amount of feature variance they explain. Keeping only the leading components can produce a lower-dimensional representation of the data.
PCA is especially useful for visualization and for summarizing correlated high-dimensional measurements, but it is unsupervised and variance-focused. Scaling choices, validation strategy, and the meaning of discarded directions all matter.
Most Important Insight to Remember
PCA does not simply “delete columns.” It rotates the feature space to find new directions of high variance, then lets us represent the data using fewer of those directions when that trade-off is useful.