A histogram of transaction amounts shows a very long right tail with a few values in the billions. What should you do before building a fraud model?

Investigate whether the extreme values are genuine (wire transfers, institutional trades) or data errors. If genuine, consider log-transforming the amount feature. If errors, correct or remove them — Blindly removing outliers can delete genuinely important data — a billion-rupee wire transfer might itself be the fraud signal. Investigate first, then decide on transformation or removal based on what you learn.

A correlation matrix shows that 'total_rooms' and 'total_bedrooms' correlate at 0.97 in a housing dataset. What does this imply?

These features carry nearly identical information. For linear models, this creates multicollinearity — inflating coefficient standard errors and making individual feature interpretations unstable — You might keep one, combine them into a ratio (bedrooms per room), or use Ridge regression which handles multicollinearity. Simply noting the correlation during EDA prevents you from later being confused by unstable linear model coefficients.

During EDA, a feature called 'days_since_last_login' is missing for 68% of users. What is the most insightful first question to ask?

Is the missingness itself informative — do users who never logged in (infinite days) have different churn rates than users who logged in recently? — 68% missing 'days since last login' likely means 68% of users have never logged in — which is a powerful signal in itself. Creating a binary 'has_never_logged_in' feature captures this signal better than any imputation strategy.

What does a Q-Q (quantile-quantile) plot reveal about a feature's distribution?

Whether the feature's distribution matches a theoretical distribution (usually normal) — points following the diagonal indicate normality; S-shaped deviations indicate heavy tails; bowed deviations indicate skewness — Many statistical tests and linear model diagnostics assume normality. A Q-Q plot quickly reveals whether this assumption holds. A feature that looks roughly normal in a histogram might reveal heavy tails in its Q-Q plot.

Exploratory Data Analysis (EDA)

Before building machine learning models or creating dashboards, analysts first need to understand the data properly. Raw datasets often contain hidden problems such as missing values, duplicate records, unusual patterns, or incorrect formatting.

EDA in Simple Words

Exploratory Data Analysis means looking at the data carefully before asking a machine learning model to learn from it. Think of it like checking a question paper before you start solving: first understand what is given, look for missing or unusual values, and notice important patterns.

Look

What rows, columns and values do we have?

Check

Are values missing, duplicated or unusual?

Discover

What patterns and relationships can we see?

A Tiny Student Example

Suppose we receive this small dataset before building a model to predict exam scores:

Student	Study Hours	Attendance	Score
A	2	70%	48
B	4	82%	65
C	6	91%	?
D	8	95%	88

Before modelling, EDA would ask:

Why is Student C's score missing?
Do higher study hours generally come with higher scores?
Is any value unusually high or low?
Are all columns stored in the correct data type?

What is Exploratory Data Analysis?

Exploratory Data Analysis (EDA) is the practice of examining a dataset before applying any machine learning algorithm — using statistics and visualisations to understand its structure, quality, and patterns. Think of EDA as the detective phase: you are not yet building a model, you are investigating the data to form hypotheses and catch problems early.

In real ML projects, careful EDA can make the difference between a reliable model and a misleading one. A model trained on poorly understood data with hidden missing values, wrong data types, or undetected outliers will perform unreliably — no matter how advanced the algorithm. EDA is the foundation everything else is built upon.

The main objectives of EDA are to understand the dataset structure and shape, detect missing or erroneous values before they corrupt training, find relationships and correlations between features, identify outliers that may skew model learning, discover hidden trends or class imbalances, and ultimately prepare the data for reliable machine learning.

Why is EDA Important?

🔍

Understand Data Structure

EDA reveals the number of rows, columns, data types, and value ranges. You cannot model what you do not understand.

🚨

Detect Missing Values

Real-world data often has gaps. EDA helps quantify how many missing values exist per column and guides your next cleaning decision.

📉

Find Outliers

Extreme values can distort model training. EDA exposes them through boxplots, histograms, and z-score analysis.

🔗

Identify Relationships

Correlation matrices and scatter plots reveal which features are predictive and which are redundant or collinear.

⚖️

Spot Class Imbalance

If 98% of your labels are "No" and 2% are "Yes", accuracy alone can be misleading. EDA catches this before you train.

💡

Guide Feature Engineering

EDA insights — like skewed distributions or interaction patterns — directly inspire which new features to create.

A Simple Numerical Summary

One of the first things EDA does is summarize a numeric column. Suppose five students scored:

40, 50, 60, 70, 80

Mean (average)

(40 + 50 + 60 + 70 + 80) ÷ 5

300 ÷ 5 = 60

Range

Highest − Lowest

80 − 40 = 40

The mean tells us the center of these scores. The range tells us how far the lowest and highest scores are apart. EDA usually combines several summaries and plots because no single number tells the whole story.

Types of Exploratory Data Analysis

Univariate

Look at: 1 variable

Example: What does the distribution of exam scores look like?

Useful visuals: histogram, box plot, bar chart

Bivariate

Look at: 2 variables

Example: Do study hours and exam scores move together?

Useful visuals: scatter plot, grouped box plot, crosstab

Multivariate

Look at: 3+ variables

Example: How do several features work together to separate classes?

Useful visuals: heatmap, pair plot, PCA plot

1. Univariate Analysis — Exploring One Variable

Univariate analysis examines each feature independently to understand its distribution, central tendency, spread, and the presence of outliers. It answers questions like: Is this feature normally distributed? Is it skewed? Are there unusual extreme values?

univariate_analysis.py

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# Example dataset: Student exam scores
scores = pd.Series([45, 52, 67, 72, 58, 89, 74, 63, 91, 55,
                    78, 83, 69, 77, 84, 61, 93, 48, 71, 86])

# --- Numerical summary ---
print("Count:   ", scores.count())
print("Mean:    ", round(scores.mean(), 2))
print("Median:  ", scores.median())
print("Std Dev: ", round(scores.std(), 2))
print("Min/Max: ", scores.min(), "/", scores.max())
print("Skewness:", round(scores.skew(), 3))  # > 0 = right skew

# --- Visualisations ---
fig, axes = plt.subplots(1, 3, figsize=(14, 4))

# Histogram — see distribution shape
axes[0].hist(scores, bins=8, color='#6366f1', edgecolor='white')
axes[0].set_title('Histogram — Distribution Shape')
axes[0].set_xlabel('Exam Score')

# Boxplot — see median, IQR, outliers
axes[1].boxplot(scores, patch_artist=True,
                boxprops=dict(facecolor='#e0e7ff'))
axes[1].set_title('Boxplot — Spread & Outliers')

# KDE — smooth distribution estimate
scores.plot.kde(ax=axes[2], color='#6366f1')
axes[2].set_title('KDE — Probability Density')

plt.tight_layout()
plt.savefig('univariate.png', dpi=120)
plt.show()

2. Bivariate Analysis — Relationships Between Two Variables

Bivariate analysis investigates how two variables relate to each other. It can reveal correlations (do study hours predict exam score?), group differences (do students in Delhi outperform those in Mumbai?), or categorical associations.

bivariate_analysis.py

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

df = pd.DataFrame({
    'Study_Hours': [2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
    'Exam_Score':  [52, 58, 64, 70, 75, 82, 88, 91, 94, 97],
    'City':        ['Delhi','Mumbai','Delhi','Pune','Mumbai',
                    'Delhi','Pune','Mumbai','Delhi','Pune'],
    'Passed':      [False, False, True, True, True,
                    True,  True,  True, True, True],
})

fig, axes = plt.subplots(1, 3, figsize=(15, 4))

# 1. Scatter plot: numerical vs numerical
axes[0].scatter(df['Study_Hours'], df['Exam_Score'], color='#6366f1', s=80)
axes[0].set_xlabel('Study Hours')
axes[0].set_ylabel('Exam Score')
axes[0].set_title('Scatter: Study Hours vs Score')

# Correlation coefficient
corr = df['Study_Hours'].corr(df['Exam_Score'])
print(f"Pearson correlation: {corr:.4f}")   # close to 1 = strong positive

# 2. Boxplot: categorical vs numerical
df.boxplot(column='Exam_Score', by='City', ax=axes[1])
axes[1].set_title('Boxplot: Score by City')
axes[1].set_xlabel('City')

# 3. Cross-tabulation: categorical vs categorical
ct = pd.crosstab(df['City'], df['Passed'])
print("
Cross-tabulation City vs Passed:")
print(ct)
ct.plot(kind='bar', ax=axes[2], color=['#ef4444','#10b981'])
axes[2].set_title('Pass Rate by City')

plt.tight_layout()
plt.savefig('bivariate.png', dpi=120)
plt.show()

Output

Pearson correlation: 0.9924   ← very strong positive relationship

Cross-tabulation City vs Passed:
Passed  False  True
City
Delhi       1     3
Mumbai      1     2
Pune        0     3

3. Multivariate Analysis — Three or More Variables

Multivariate analysis examines how multiple features interact simultaneously. It answers questions like: Which combination of features best separates classes? Are any features so correlated that they are redundant? It forms the basis of feature selection and dimensionality reduction.

multivariate_analysis.py

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris

iris = load_iris()
df = pd.DataFrame(iris.data, columns=iris.feature_names)
df['species'] = pd.Categorical.from_codes(iris.target, iris.target_names)

# 1. Correlation heatmap — detect redundant features
plt.figure(figsize=(7, 5))
corr_matrix = df.drop(columns='species').corr()
sns.heatmap(corr_matrix, annot=True, fmt='.2f', cmap='RdYlGn',
            center=0, square=True, linewidths=0.5)
plt.title('Feature Correlation Heatmap')
plt.savefig('heatmap.png', dpi=120)
plt.show()
print("Correlation matrix:")
print(corr_matrix.round(2))

# 2. Pair plot — all feature combinations coloured by class
sns.pairplot(df, hue='species', palette='husl', plot_kws={'alpha':0.6})
plt.suptitle('Pair Plot — All Feature Combinations', y=1.02)
plt.savefig('pairplot.png', dpi=120)
plt.show()

Output

Correlation matrix:
                     sepal length  sepal width  petal length  petal width
sepal length (cm)           1.00        -0.12          0.87         0.82
sepal width (cm)           -0.12         1.00         -0.43        -0.37
petal length (cm)           0.87        -0.43          1.00         0.96
petal width (cm)            0.82        -0.37          0.96         1.00

Insight: petal length and petal width are highly correlated (0.96)
→ one of them may be redundant for modelling

Common Steps in Exploratory Data Analysis

An effective EDA process usually follows a repeatable set of checks. The exact order can vary by dataset, but this beginner-friendly sequence is a good starting point.

Step 1 – Import Required Libraries

Python provides powerful libraries for data analysis and visualization. Common libraries include Pandas, NumPy, Matplotlib, Seaborn, and Plotly.

1_import_libraries.py

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

Step 2 – Load the Dataset

Datasets are usually stored in CSV, Excel, or database formats.

2_load_data.py

df = pd.read_csv("data.csv")
print(df.head())

Output

      Name   Age  Salary  Experience
0    Alice  28.0   60000           3
1      Bob  34.0   75000           6
2  Charlie   NaN   50000           1
3    David  45.0  120000          15
4      Eve  30.0   80000           5

Step 3 – Understand Dataset Structure

3_structure.py

# Check Shape
print(df.shape)

# Check Data Types
df.info()

# Statistical Summary
print(df.describe())

Output

(100, 4)

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 100 entries, 0 to 99
Data columns (total 4 columns):
 #   Column      Non-Null Count  Dtype  
---  ------      --------------  -----  
 0   Name        100 non-null    object 
 1   Age         92 non-null     float64
 2   Salary      100 non-null    int64  
 3   Experience  100 non-null    int64  
dtypes: float64(1), int64(2), object(1)
memory usage: 3.2+ KB

              Age         Salary  Experience
count   92.000000     100.000000  100.000000
mean    35.240000   72500.500000    7.200000
std      8.120000   24000.100000    4.500000
min     22.000000   45000.000000    1.000000
25%     29.000000   55000.000000    3.000000
50%     34.000000   68000.000000    6.000000
75%     41.000000   85000.000000   10.000000
max     60.000000  150000.000000   25.000000

Step 4 – Handling Missing Values

4_missing_values.py

# Check Missing Values
print(df.isnull().sum())

# Remove Missing Values (Optional, drops rows with missing data)
# df.dropna()

# Fill Missing Values with Mean
df.fillna(df.mean(numeric_only=True), inplace=True)

Output

Name          0
Age           8
Salary        0
Experience    0
dtype: int64

Important: filling every missing numeric value with the mean is only a simple demonstration. In a real project, first investigate why values are missing and whether mean, median, mode, model-based imputation, or leaving the value missing is more appropriate.

Step 5 – Detecting Duplicate Records

Duplicate data can create misleading analysis.

5_duplicates.py

# Count duplicates
print(df.duplicated().sum())

# Remove duplicates
df.drop_duplicates(inplace=True)

Output

Data Visualization in EDA

Visualization makes analysis easier and faster. The best chart depends on the question you want to answer.

Question	Simple Visual	Example
How is one numeric variable distributed?	Histogram	Age distribution
Are there unusual extreme values?	Box plot	Very high salary
Do two numeric variables move together?	Scatter plot	Experience vs salary
Which numeric features are related?	Correlation heatmap	Age, salary, experience

# Histogram Example
plt.hist(df['Age'])
plt.show()

Output

# Box Plot Example
sns.boxplot(x=df['Salary'])
plt.show()

Output

# Scatter Plot Example
sns.scatterplot(
    x='Experience', y='Salary', data=df
)
plt.show()

Output

# Correlation Heatmap Example
plt.figure(figsize=(10,6))
sns.heatmap(df.corr(numeric_only=True), annot=True)
plt.show()

Output

Best Practices & Challenges

Best Practices for EDA

Always understand business objectives first
Clean the data carefully
Use multiple visualizations
Validate unusual patterns
Document observations

Challenges in EDA

Missing Data
Outliers
Large Datasets
Data Imbalance
Inconsistent Formats

The Garbage In, Garbage Out Warning

Machine learning models learn patterns from the values they are given. If a missing age is stored as -999 and the model is not told that this value means "missing", it may treat -999 as a real age and learn a misleading pattern. EDA helps catch problems like this before training.

Common EDA Questions

Do we clean the data before or during EDA?

Usually both. Initial EDA first reveals missing values, duplicates and suspicious values. You then clean carefully and explore again to confirm that the dataset now makes sense.

Does correlation prove that one feature causes another?

No. Correlation means two variables move together in the observed data. It does not by itself prove that one causes the other.

Should every outlier be removed?

No. First investigate it. A value may be a typing error, a measurement problem, or a genuine rare case that is important to keep.

Continue learning

After EDA, the next common tasks are handling missing data, feature engineering, and choosing clear data visualizations for ML.

Practice Checkpoint

Test Your Recognition

Scenario:

You are preparing a dataset of 10,000 employees to predict salary. You want to quickly check if someone accidentally typed an extra '0' for someone's salary, creating an extreme anomaly that could break your linear regression model.

Which simple EDA visualization is especially useful for quickly spotting potential extreme values?

Reveal Answer

Answer: A Box Plot.

A box plot summarizes the center and spread of a numeric variable using quartiles and whiskers. Values beyond the whiskers are often shown as separate points, making them easy to investigate as potential outliers. A flagged point is not automatically an error—it may be a genuine extreme value.

Exploratory Data Analysis (EDA)

EDA in Simple Words

A Tiny Student Example

What is Exploratory Data Analysis?

Why is EDA Important?

A Simple Numerical Summary

Types of Exploratory Data Analysis

1. Univariate Analysis — Exploring One Variable

2. Bivariate Analysis — Relationships Between Two Variables

3. Multivariate Analysis — Three or More Variables

Common Steps in Exploratory Data Analysis

Step 1 – Import Required Libraries

Step 2 – Load the Dataset

Step 3 – Understand Dataset Structure

Step 4 – Handling Missing Values

Step 5 – Detecting Duplicate Records

Data Visualization in EDA

Best Practices & Challenges

Best Practices for EDA

Challenges in EDA

Common EDA Questions

Practice Checkpoint

Test Your Recognition

Apply this concept in a real hands-on project

Finished "Exploratory Data Analysis (EDA)"?

Was this tutorial helpful?

Summary / Key Takeaways

Knowledge Check

Knowledge Check Review & Explanations

Continue Learning

Related topics

Free Download: ML Interview Cheatsheet

Optional Learning Resources

Hands-On Machine Learning