Handling Missing Data

Handling Missing Values

Missing data means some information is absent from a dataset. Before training a model, we should first understand what is missing, why it may be missing, and how our chosen treatment could change the data.

What are Missing Values?

Missing values are data entries that are not stored properly inside a dataset. In Python and data science libraries, missing values are commonly represented as:

• NaN (Not a Number)
• NULL
• None
• Empty cells
• Unknown values

Missing Data in Simple Words

Think of a school record where one student forgot to enter their age and another student's marks were not recorded. The empty cells are the missing values.

Why Handling Missing Values is Important

1. Supports Reliable Models: Missing values can change the information available to a model, so they should be handled deliberately.
2. Prevents Bias: Ignoring missing values improperly may create biased analysis.
3. Avoids Compatibility Problems: Many estimators require missing values to be handled first, although some modern algorithms can work with missing values directly.
4. Preserves Data Quality: Handling missing values properly keeps the dataset reliable.
5. Helps Better Decision Making: Businesses rely on accurate data for forecasting and analytics.

Common Reasons & Types of Missing Values

Missing data can result from human errors, technical problems, privacy concerns, data merging issues, or data corruption.

Check Missing Values with Python

Let's create a sample dataset and see how to check for missing values using pandas.

import pandas as pd
import numpy as np

data = {
    'Name': ['Aman', 'Priya', 'Rohit', 'Sneha'],
    'Age': [22, np.nan, 25, 28],
    'Salary': [45000, 52000, np.nan, 61000]
}

df = pd.DataFrame(data)
print("--- Dataset ---")
print(df)

print("\n--- Count Missing Values ---")
print(df.isnull().sum())

print("\n--- Dataset Information ---")
df.info()

--- Dataset ---
    Name   Age   Salary
0   Aman  22.0  45000.0
1  Priya   NaN  52000.0
2  Rohit  25.0      NaN
3  Sneha  28.0  61000.0

--- Count Missing Values ---
Name      0
Age       1
Salary    1
dtype: int64

--- Dataset Information ---
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 4 entries, 0 to 3
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   Name    4 non-null      object 
 1   Age     3 non-null      float64
 2   Salary  3 non-null      float64
dtypes: float64(2), object(1)
memory usage: ...

Before Choosing a Method: Measure the Missingness

First count how much data is missing. A column with 1% missing values is a different problem from a column with 70% missing values.

missing_pct = df.isna().mean() * 100
print(missing_pct.round(1))

Name       0.0
Age       25.0
Salary    25.0
dtype: float64

In this four-row example, one missing value means 1 ÷ 4 × 100 = 25%. The percentage is useful, but the reason for the missing values is just as important.

Methods to Handle Missing Values

There is no single perfect method for handling missing data. The best technique depends on dataset size, the amount and pattern of missing data, the meaning of the variable, and the problem you are solving.

1. Removing Missing Values

You can remove rows or columns containing missing values, but only when the information loss is acceptable and the missingness pattern does not create unwanted bias.

• Advantages: Simple implementation, Quick cleaning
• Disadvantages: Loss of valuable data, Smaller dataset

# Remove Rows with Missing Values:
df_dropped_rows = df.dropna()
print("Dropped Rows:\n", df_dropped_rows)

# Remove Columns with Missing Values:
df_dropped_cols = df.dropna(axis=1)
print("\nDropped Columns:\n", df_dropped_cols)

Dropped Rows:
     Name   Age   Salary
0   Aman  22.0  45000.0
3  Sneha  28.0  61000.0

Dropped Columns:
     Name
0   Aman
1  Priya
2  Rohit
3  Sneha

2. Mean Imputation

Replace missing numerical values using the average value.

• Advantages: Easy to implement, Maintains dataset size
• Disadvantages: Sensitive to outliers

df_mean = df.copy()
df_mean['Salary'] = df_mean['Salary'].fillna(df_mean['Salary'].mean())
print(df_mean)

    Name   Age        Salary
0   Aman  22.0  45000.000000
1  Priya   NaN  52000.000000
2  Rohit  25.0  52666.666667
3  Sneha  28.0  61000.000000

3. Median Imputation

Replace missing values using the median.

• Advantages: Handles outliers better, More robust than mean

df_median = df.copy()
df_median['Salary'] = df_median['Salary'].fillna(df_median['Salary'].median())
print(df_median)

    Name   Age   Salary
0   Aman  22.0  45000.0
1  Priya   NaN  52000.0
2  Rohit  25.0  52000.0
3  Sneha  28.0  61000.0

4. Other Imputation Techniques

Beyond mean and median imputation, several smarter techniques handle special cases — categorical variables, time-series data, and trending signals.

# Mode imputation — fills missing City with most common city
df['City'] = df['City'].fillna(df['City'].mode()[0])

# Example:
# Before: ['Delhi', NaN, 'Mumbai', 'Delhi', NaN]
# mode()[0] = 'Delhi'
# After:  ['Delhi', 'Delhi', 'Mumbai', 'Delhi', 'Delhi']

# Forward fill — use previous row's value
df = df.ffill()

# Example — Daily temperature readings:
# Before: [22, NaN, NaN, 25, NaN]
# After:  [22,  22,  22, 25,  25]
# Explanation: NaN rows inherit the last valid reading above them

# Backward fill — use next row's value
df = df.bfill()

# Example — Daily temperature readings:
# Before: [NaN, NaN, 22, 25, NaN]
# After:  [ 22,  22, 22, 25, NaN]  ← last NaN stays (no future value)
# Explanation: NaN rows inherit the nearest valid reading below them

# Linear interpolation — estimates a value between two known points
df['Salary'] = df['Salary'].interpolate(method='linear')

# Example — Annual Salary (₹ in lakhs):
# Index:  0     1     2     3      4
# Before: [4.0, NaN,  NaN,  NaN,  8.0]
# After:  [4.0, 5.0,  6.0,  7.0,  8.0]
# Explanation: Linear interpolation divides the gap (8.0-4.0)/4 = 1.0
#              and adds 1.0 at each step → perfectly smooth ramp

# Other interpolation methods:
df['Salary'].interpolate(method='polynomial', order=2)  # curved trend
df['Salary'].interpolate(method='time')                 # for DatetimeIndex

5. Advanced Techniques

KNN Imputation: K-Nearest Neighbors predicts missing values using similar records. It can use relationships between variables, but it is not guaranteed to outperform simple methods and can be more computationally expensive. Feature scales and the choice of neighbors also matter.

Predictive Model Imputation: Machine learning models can estimate missing values. Examples include regression-based methods and iterative multivariate imputation. These methods need careful validation because a more complex imputer is not automatically a better one.

6. Missing-Value Indicator

Sometimes the fact that a value is missing is itself useful information. You can add a simple 0/1 column before imputation.

This can help a model distinguish an originally observed value from one that was filled later. Whether it helps should be checked during validation.

Choosing the Right Missing Value Technique

Complete Python Example

import pandas as pd
import numpy as np

data = {
    'Age': [22, 25, np.nan, 30, 28],
    'Salary': [45000, np.nan, 52000, 61000, np.nan]
}

df = pd.DataFrame(data)
print("--- Missing Values Count ---")
print(df.isnull().sum())

# Impute Operations
df['Age'] = df['Age'].fillna(df['Age'].median())
df['Salary'] = df['Salary'].fillna(df['Salary'].mean())

print("\n--- Dataset After Handling Missing Values ---")
print(df)

--- Missing Values Count ---
Age       1
Salary    2
dtype: int64

--- Dataset After Handling Missing Values ---
    Age        Salary
0  22.0  45000.000000
1  25.0  52666.666667
2  26.5  52000.000000
3  30.0  61000.000000
4  28.0  52666.666667

Best Practices & Common Mistakes

Common Questions