Feature Scaling

Feature Scaling

Feature Scaling changes the numerical scale of features so that features measured in very different units can be compared more fairly by scale-sensitive machine learning methods.

Feature Scaling in Simple Words

Imagine two columns in the same dataset: Age and Salary. Age may contain values such as 20, 35 and 50, while Salary may contain values such as 300,000 and 900,000. A large salary number does not automatically mean Salary is more important. It is simply measured using a much larger numerical unit.

Important: Scaling does not decide which feature is actually useful. It only changes the numerical representation so that units such as rupees, years and centimetres do not unfairly control a calculation.

What is Feature Scaling?

Feature Scaling is the process of transforming numerical features into a similar range so that no feature dominates others during model training.

Suppose a dataset contains:

For algorithms that depend on distances, gradients, dot products or variance, the Salary feature can dominate the numerical calculation simply because its values are much larger.

Why Feature Scaling is Important

Many algorithms calculate:

• Distance
• Gradients
• Variance
• Similarity
• Optimization paths

If one feature has much larger numerical values, a scale-sensitive algorithm may give it disproportionate influence even when that feature is not inherently more informative.

Algorithms That Need Feature Scaling

1. Min-Max Scaling (Often Called Normalization)

Min-Max Scaling rescales values into a chosen fixed range such as [0, 1] or [-1, 1].

Simple example: Suppose Age ranges from 18 to 60 and we want to scale Age = 30.

from sklearn.preprocessing import MinMaxScaler
import pandas as pd

data = pd.DataFrame({'Age': [18, 25, 40, 60]})

scaler = MinMaxScaler()
scaled = scaler.fit_transform(data)

print(scaled)

[[0.        ]
 [0.16666667]
 [0.52380952]
 [1.        ]]

2. Standardization (Z-Score Scaling)

Standardization transforms data so that its Mean = 0 and Standard deviation = 1.

Simple example: Mean = 50, standard deviation = 10 and the value is 70.

from sklearn.preprocessing import StandardScaler
import pandas as pd

data = pd.DataFrame({'Salary': [20000, 30000, 40000, 50000]})

scaler = StandardScaler()
scaled = scaler.fit_transform(data)

print(scaled)

[[-1.34164079]
 [-0.4472136 ]
 [ 0.4472136 ]
 [ 1.34164079]]

Standardization is commonly useful for SVM, PCA, many linear/logistic models, neural networks and other scale-sensitive methods. Because the mean and standard deviation themselves can be influenced by extreme values, StandardScaler is still sensitive to outliers.

3. Robust Scaling

Robust Scaling is designed to handle outliers better. Instead of the Mean and Standard Deviation, it uses the Median and Interquartile Range (IQR).

Extreme values can strongly affect the mean and standard deviation. The median and IQR are usually less influenced by extreme values, which is why RobustScaler can be useful for outlier-heavy features. Let's see a simple example:

from sklearn.preprocessing import RobustScaler
import pandas as pd

# Notice the very large 10,000,000 value
data = pd.DataFrame({'Income': [25000, 30000, 35000, 10000000]})

scaler = RobustScaler()
scaled = scaler.fit_transform(data)

print(scaled)

[[-0.003003  ]
 [-0.001001  ]
 [ 0.001001  ]
 [ 3.99099099]]

4. Unit Vector Scaling

Unit-vector scaling (normalization by vector norm) rescales each sample so that its vector length becomes 1. It is especially useful when the direction of a vector matters more than its magnitude, such as many text and cosine-similarity tasks.

Simple example: Take the vector [3, 4].

from sklearn.preprocessing import Normalizer

X = [[3, 4]]
normalizer = Normalizer()
print(normalizer.fit_transform(X))

[[0.6 0.8]]

Feature Scaling vs Normalization vs Standardization

Why Gradient Descent Improves with Scaling

Pipelines in Scikit-Learn

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('model', LogisticRegression())
])

# Use pipeline.fit(X_train, y_train)

scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)

Which Scaling Method Should I Start With?

This is a starting guide, not a universal rule. The best preprocessing choice depends on the algorithm, distribution and evaluation results.

Common Questions About Feature Scaling

Interview Knowledge Test