Feature Selection & Extraction

Feature Selection and Feature Extraction in Machine Learning

Feature selection keeps the most useful original input features, while feature extraction transforms the original inputs into a smaller set of new features. Both approaches can reduce dimensionality, simplify models, lower computation, and sometimes improve generalization.

In simple terms, feature selection asks “which original features should I keep?” while feature extraction asks “how can I represent the same information with fewer new features?” This guide explains both approaches, when to use each one, and the most common methods used in Machine Learning.

Feature Selection & Extraction in Simple Words

Imagine your dataset has many columns, but not every column is equally useful. You have two simple choices: keep only useful original columns, or combine information into a smaller set of new columns.

What is a Feature?

A feature is an input variable used by a Machine Learning model. Features help models discover patterns.

Why Feature Optimization Matters

Real-world datasets often contain problems that reduce model performance:

• Irrelevant Features: E.g., Random IDs
• Duplicate Features: E.g., Same information repeated
• Noisy Features: E.g., Corrupted sensor data
• High-Dimensional Data: E.g., Thousands of columns

What is Feature Selection?

Feature Selection is the process of selecting the most useful features and removing unnecessary ones. Instead of creating new features, it chooses the best subset from existing columns.

Types of Feature Selection Techniques

1. Filter Methods

Filter methods score or remove features using statistical properties before fitting the final predictive model. They are often fast and scalable, but a simple one-feature-at-a-time test may miss useful interactions between features.

• Correlation: Highly correlated input features can carry very similar information. For some models, you may consider keeping one of them after checking domain meaning and validation performance.
• Chi-Square Test: For classification, it can test whether non-negative input features are related to the class label.
• ANOVA F-test: In supervised feature selection, it can score how strongly a numerical feature differs across target classes.
• Mutual Information: Scores how much knowing a feature reduces uncertainty about the target. It can capture some non-linear relationships that simple correlation may miss.
• Variance Threshold: Remove low-variance features.

from sklearn.datasets import make_classification
from sklearn.feature_selection import SelectKBest, mutual_info_classif

X, y = make_classification(
    n_samples=200,
    n_features=10,
    n_informative=5,
    random_state=42
)

selector = SelectKBest(score_func=mutual_info_classif, k=5)
X_selected = selector.fit_transform(X, y)

print("Original shape:", X.shape)
print("Selected shape:", X_selected.shape)
print("Selected columns:", selector.get_support(indices=True))

Original shape: (200, 10)
Selected shape: (200, 5)
Selected columns: [0 2 4 6 9]

# The exact selected indexes can vary because mutual information is estimated.

from sklearn.feature_selection import VarianceThreshold
import pandas as pd

# Creating a dataset where feature 'B' has very low variance (mostly 1s)
data = pd.DataFrame({
    'A': [1, 2, 3, 4, 5],
    'B': [1, 1, 1, 1, 0],
    'C': [5, 4, 3, 2, 1]
})

selector = VarianceThreshold(threshold=0.2)
X_new = selector.fit_transform(data)

print(X_new)

[[1 5]
 [2 4]
 [3 3]
 [4 2]
 [5 1]]
# Column B is removed because its variance is below 0.2.

2. Wrapper Methods

Wrapper methods evaluate multiple feature combinations using actual ML model performance. Examples include Forward Selection, Backward Selection, and Recursive Feature Elimination (RFE).

Recursive Feature Elimination (RFE) repeatedly trains a estimator and recursively removes the least important features until the requested number remains. Because it repeatedly fits a model, it can be more computationally expensive than simple filters.

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=100, n_features=10, random_state=42)

model = LogisticRegression(max_iter=1000)
# Select top 5 features out of 10
rfe = RFE(model, n_features_to_select=5)
X_rfe = rfe.fit_transform(X, y)

print("Original shape:", X.shape)
print("New shape:", X_rfe.shape)
print("Which columns were selected?", rfe.support_)

Original shape: (100, 10)
New shape: (100, 5)
Which columns were selected? [ True  True  True  True False False  True False False False]

3. Embedded Methods

Embedded methods use information learned while fitting a model to decide which features matter. Examples include Lasso-based selection and importance-based selection with suitable tree models.

Lasso Regression uses an L1 penalty and can drive some coefficients exactly to zero. Those zero-coefficient features can then be treated as unselected. Ridge, by contrast, usually shrinks coefficients but does not normally create sparse zero-coefficient feature sets, so it is not usually used as a direct selector.

What is Feature Extraction?

Feature Extraction transforms the original variables into a new representation. Unlike selection, which keeps a subset of original columns, extraction creates new features that can summarize important patterns in the original data.

Why Feature Extraction Matters: It can reduce dimensionality, make some datasets easier to visualize, and sometimes reduce computation. The trade-off is that the new components may be harder to interpret and some information can be lost.

Core Techniques in Feature Extraction

from sklearn.decomposition import PCA
import numpy as np

# A small fixed dataset with 4 numeric features
X = np.array([
    [1, 2, 1, 4],
    [2, 4, 2, 3],
    [3, 5, 3, 5],
    [4, 7, 4, 6],
    [5, 8, 6, 7],
    [6, 10, 5, 9],
    [7, 11, 7, 8],
    [8, 13, 9, 11]
], dtype=float)

# Reduce 4 original features to 2 new components
pca = PCA(n_components=2)
X_compressed = pca.fit_transform(X)

print("Original Data Shape:", X.shape)
print("Compressed Shape:", X_compressed.shape)
print("Variance Captured (%):", np.round(pca.explained_variance_ratio_ * 100, 2))
print("Total captured (%):", round(pca.explained_variance_ratio_.sum() * 100, 2))

Original Data Shape: (8, 4)
Compressed Shape: (8, 2)
Variance Captured (%): [97.31  1.76]
Total captured (%): 99.06

Pros and Cons of Extraction

Feature Selection vs Feature Extraction in Machine Learning

Since Feature Selection and Feature Extraction are related but serve distinctly different purposes, let’s quickly contrast them across key aspects for a better understanding:

When to use Selection: Need interpretability, want faster models, features are redundant, working with tabular data.
When to use Extraction: High-dimensional data exists, need compression, working with images/text/audio, visualization is needed.

The Curse of Dimensionality

As dimensionality grows, data can become sparse and distance-based patterns may become harder to estimate reliably. More features can also increase computation and overfitting risk. Feature selection or extraction can help, but they should be validated rather than applied automatically.

Common Questions

Interview Knowledge Test