Encoding Categorical Data

Encoding Categorical Data

Many Machine Learning algorithms expect numerical input. If a dataset contains categories such as Red, Blue,Delhi, or High, we often need to represent those categories numerically before training the model. This process is called categorical encoding.

Categorical encoding is an important preprocessing technique in Machine Learning. The correct method depends on what the category means. For example, Red / Blue / Green have no natural order, whileLow / Medium / High do have a meaningful order.

This guide explains categorical encoding from beginner to advanced level using simple examples, tables, Scikit-learn implementations, and practical best practices.

Encoding in Simple Words

Imagine a school dataset with a column called T-shirt Size. A model may not be able to use the words S, M, andL directly. Encoding changes those categories into a numerical form while trying to preserve their meaning.

What is Categorical Data?

Categorical data represents labels, groups, or categories rather than measurable numeric quantities. Unlike numerical features like salary or age, categorical variables describe qualities or classes.

Why Machine Learning Models Need Encoding

Many commonly used ML estimators expect numerical feature arrays. Text categories such as Red, Blue, or Green therefore need a suitable numerical representation before those estimators can use them. The representation matters because different algorithms use the numbers in different ways.

Types of Categorical Variables

Major Encoding Techniques

1. Label Encoding

In Scikit-learn, LabelEncoder is designed for the target variable y. For example, a classification target containing Fail and Passcan be converted to 0 and 1. The model still treats these as class labels, not as measurements where one class is "twice" another.

from sklearn.preprocessing import LabelEncoder

y = ['Pass', 'Fail', 'Pass', 'Pass']

encoder = LabelEncoder()
encoded_y = encoder.fit_transform(y)

print("Original:", y)
print("Encoded: ", encoded_y)
print("Classes: ", encoder.classes_)

Original: ['Pass', 'Fail', 'Pass', 'Pass']
Encoded:  [1 0 1 1]
Classes:  ['Fail' 'Pass']

Why not simply write Red = 1, Blue = 2, Green = 3?

Those numbers accidentally create an order and distance that the colors do not have. A distance-based or linear model may treat Green (3) as farther from Red (1) than Blue (2), even though that relationship was invented by us.

2. One-Hot Encoding (Dummy Variables)

For Nominal Data (no order), we use One-Hot Encoding. Instead of assigning ascending numbers in a single column, we create entirely new columns for every single unique category. We use1 to indicate "Hot" (True) and 0 for "Cold" (False).

This is powerful because it removes any fake ordering between categories. The model no longer assumes Green > Blue > Red. Each category becomes completely independent.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame({'Color': ['Red', 'Blue', 'Green']})

# Pandas Method
encoded_pd = pd.get_dummies(df['Color'], dtype=int)
print("Pandas get_dummies:\n", encoded_pd, "\n")

# Scikit-Learn Method
encoder = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
result = encoder.fit_transform(df[['Color']])
print("Scikit-Learn OneHotEncoder:\n", result)

Pandas get_dummies:
    Blue  Green  Red
0     0      0    1
1     1      0    0
2     0      1    0 

Scikit-Learn OneHotEncoder:
 [[0. 0. 1.]
 [1. 0. 0.]
 [0. 1. 0.]]

pd.get_dummies(df, drop_first=True)
# or
OneHotEncoder(drop='first')

3. Ordinal Encoding

Ordinal Encoding is for input features that truly have an order. A familiar example is T-shirt size: Small comes before Medium, and Medium comes before Large.

from sklearn.preprocessing import OrdinalEncoder

data = [['Small'], ['Medium'], ['Large']]

# Tell the encoder the real order
encoder = OrdinalEncoder(categories=[['Small', 'Medium', 'Large']])

encoded = encoder.fit_transform(data)
print(encoded)

[[0.]
 [1.]
 [2.]]

4. Binary Encoding

Binary Encoding first converts categories into integers and then into binary format. One-Hot Encoding becomes inefficient for "high-cardinality" features (too many unique categories). Binary Encoding heavily reduces dimensionality.

import category_encoders as ce
import pandas as pd

df = pd.DataFrame({'Color': ['Red', 'Blue', 'Green']})

encoder = ce.BinaryEncoder(cols=['Color'])
result = encoder.fit_transform(df)

print(result)

   Color_0  Color_1
0        0        1
1        1        0
2        1        1

5. Frequency Encoding

Categories are replaced with their occurrence frequency. It is efficient for large datasets but causes information loss if categories share the same frequency.

Frequency encoding is compact, but two different categories with the same count receive the same number. It also makes sense only when category frequency carries useful information for the problem.

import pandas as pd

data = pd.Series(['Red', 'Blue', 'Red', 'Green', 'Red', 'Blue'])

freq = data.value_counts()
print("Frequencies:\n", freq)

# Map the frequencies back to original data
encoded = data.map(freq)
print("\nEncoded Data:\n", encoded.to_numpy())

Frequencies:
 Red      3
Blue     2
Green    1
dtype: int64

Encoded Data:
 [3 2 3 1 3 2]

6. Target Encoding

Target Encoding replaces a category with a number learned from the target. For a binary target such as Purchased = 1 andNot Purchased = 0, the basic intuition is the average target value for each category.

import pandas as pd
from sklearn.preprocessing import TargetEncoder

df = pd.DataFrame({
    'City': ['Delhi', 'Mumbai', 'Delhi', 'Chennai', 'Mumbai'],
    'Purchased': [1, 0, 1, 1, 0]
})

encoder = TargetEncoder(cv=2)

# fit_transform uses cross-fitting on the training data
encoded_city = encoder.fit_transform(df[['City']], df['Purchased'])

print(encoded_city.shape)

(5, 1)
# The encoded values are learned using cross-fitting and smoothing.

Target Encoding is an advanced choice. It can be useful for high-cardinality features, but it should normally be evaluated inside a proper validation or cross-validation workflow.

Pipelines & ColumnTransformer

Real datasets contain a mix of Numerical and Categorical features. We can preprocess them together cleanly using ColumnTransformer.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

categorical_cols = ['Gender', 'City']

# Define the preprocessing steps
preprocessor = ColumnTransformer(
    transformers=[
        ('cat', OneHotEncoder(handle_unknown='ignore'), categorical_cols)
    ]
)

# Bundle preprocessing and modeling into a Pipeline
pipeline = Pipeline([
    ('preprocessor', preprocessor),
    ('model', LogisticRegression())
])

# Now you can just use pipeline.fit(X_train, y_train)

The High Cardinality Problem

High Cardinality means that a feature has many unique categories—for example thousands of product codes or locations. One-hot encoding can then create a very wide sparse feature matrix, which may increase memory use, computation time, and overfitting risk.

Possible alternatives: group rare categories, use frequency/count encoding, binary or hashing-style encoders, carefully cross-fitted target encoding, native categorical support in an appropriate model, or learned embeddings in deep learning. The best choice depends on the data and model.

Real-World Applications

Common Beginner Mistakes

• ❌ Using LabelEncoder on ordinary nominal input features.
• ❌ Automatically one-hot encoding extremely high-cardinality columns without checking the resulting feature width and model requirements.
• ❌ Using different encoders for training and testing data (Always use fit_transform on train, and ONLY transform on test).
• ❌ Ignoring unseen categories during prediction instead of deciding how the encoder should handle them.
• ❌ Allowing data leakage when using Target Encoding.
• ❌ Treating identifier columns such as customer IDs as ordinary categories without asking whether they should be used at all.

Practice Checkpoint

Common Questions About Categorical Encoding