What is the difference between df.loc and df.iloc?

loc selects by label (index name or column name); iloc selects by integer position (0-based row and column numbers) — df.loc['alice', 'salary'] selects the row labelled 'alice' and the column named 'salary'. df.iloc[0, 3] selects row 0 and column 3 by position. Mixing them up is one of the most common pandas bugs.

What does df.groupby('city')['sales'].mean() return?

A Series with one entry per unique city, containing the mean sales for customers in that city — groupby splits the DataFrame into groups by the specified column, then applies the aggregation (mean) to each group independently. This is the standard pandas pattern for computing segment-level statistics.

What is the risk of using df.dropna() by default on a dataset with many features?

If each of 50 features has a few missing values scattered independently, dropna() may remove most or all rows — dropping all rows with any missing value can silently destroy most of your dataset — dropna() removes any row containing at least one NaN by default. With 50 features each having 5% missing values (independent), roughly 92% of rows will have at least one NaN — dropna() would discard 92% of your data silently.

What is the dummy variable trap that pd.get_dummies() can create, and how do you avoid it?

When one-hot encoding creates n binary columns that always sum to exactly 1, they are perfectly linearly dependent — this breaks linear regression's matrix inversion. Avoid by using drop_first=True to remove one reference category — Perfect multicollinearity (X'X is singular) makes the normal equation unsolvable. With cities [Mumbai, Delhi, Bengaluru], you need only 2 indicator columns — the third is always derivable from the other two.

Pandas Essentials for ML | LearnMLAcademy

Pandas for Machine Learning

Pandas is the go-to library for loading, exploring, cleaning, and transforming tabular data before feeding it into ML models. If NumPy is the engine, Pandas is the workshop where raw data gets shaped into model-ready form.

📋

DataFrame

Labelled 2D table like Excel

📈

Series

1D labelled array (single column)

🧹

Data Cleaning

Handle NaN, duplicates, types

🔍

EDA

describe(), groupby(), value_counts()

Pandas in Simple Words

Think of Pandas as an Excel-like table that you can control with Python. In machine learning, we often receive raw data as rows and columns, then use Pandas to inspect it, clean it, create useful features, and pass the prepared data to a model.

Student	Study Hours	Marks	Passed
Asha	2	48	No
Ravi	4	72	Yes
Meera	5	84	Yes

STEP 1

Load

Read CSV / Excel / database data

STEP 2

Inspect

See rows, columns, types and missing values

STEP 3

Clean

Fix missing, duplicate or incorrect values

STEP 4

Prepare

Create features and send data to the ML model

Python

pip install pandas

import pandas as pd
import numpy as np
print(pd.__version__)   # your installed pandas version

📋

1. Creating & Loading DataFrames

A DataFrame is pandas' core structure — a table with labelled rows and columns

See a DataFrame before writing code

Rows are individual records. Columns describe the information stored for every record. The left-most numbers are the index.

Index	Name	Age	Score
0	Alice	24	88
1	Bob	32	75
2	Charlie	28	92

Shape = (3, 3) here because there are 3 data rows and 3 data columns. The index is a label, not one of these three data columns.

Python

# Create DataFrame from dictionary — columns are keys
data = {
    'Name':    ['Alice', 'Bob', 'Charlie', 'Diana', 'Eve'],
    'Age':     [24, 32, 28, 35, 22],
    'Score':   [88, 75, 92, 68, 95],
    'Passed':  [True, True, True, False, True],
}
df = pd.DataFrame(data)
print(df)

Output

      Name  Age  Score  Passed
0    Alice   24     88    True
1      Bob   32     75    True
2  Charlie   28     92    True
3    Diana   35     68   False
4      Eve   22     95    True

Python

# Load from CSV — the most common way in real projects
df = pd.read_csv('students.csv')

# Other common formats:
df_excel = pd.read_excel('data.xlsx')
df_json  = pd.read_json('data.json')
df_sql   = pd.read_sql('SELECT * FROM students', connection)

# Save back to CSV
df.to_csv('cleaned_data.csv', index=False)

🔍

2. Exploring Your Dataset — Always Do This First

Before any modelling, understand your data shape, types, and quality

CHECK 1

How big?

df.shape

CHECK 2

Which columns?

df.columns

CHECK 3

Which types?

df.dtypes

CHECK 4

Anything missing?

df.isnull().sum()

CHECK 5

What do values look like?

df.describe()

Python

df = pd.DataFrame({
    'Name':   ['Alice','Bob','Charlie','Diana','Eve'],
    'Age':    [24, 32, 28, None, 22],
    'Score':  [88, 75, 92, 68, 95],
    'City':   ['Delhi','Mumbai','Delhi','Pune','Mumbai'],
})

print(df.shape)         # (5, 4) — rows × columns
print(df.dtypes)        # data type of each column
print(df.columns)       # column names

Output

(5, 4)
Name      object
Age      float64
Score      int64
City      object
dtype: object
Index(['Name', 'Age', 'Score', 'City'], dtype='object')

Python

print(df.head(3))    # first 3 rows
print(df.tail(2))    # last 2 rows
df.info()            # dtypes + non-null counts
print(df.describe()) # stats for numeric columns

Output

      Name   Age  Score    City
0    Alice  24.0     88   Delhi
1      Bob  32.0     75  Mumbai
2  Charlie  28.0     92   Delhi

     Name   Age  Score    City
3   Diana   NaN     68    Pune
4     Eve  22.0     95  Mumbai

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 4 columns):
 #   Column  Non-Null Count  Dtype
---  ------  --------------  -----
 0   Name    5 non-null      object
 1   Age     4 non-null      float64   ← 1 missing!
 2   Score   5 non-null      int64
 3   City    5 non-null      object

        Age      Score
count  4.00   5.000000
mean  26.50  83.600000
std    4.43  11.501449
min   22.00  68.000000
25%   23.50  75.000000
50%   26.00  88.000000
75%   29.00  92.000000
max   32.00  95.000000

Python

# Check missing values — critical before ML
print(df.isnull().sum())       # count NaN per column
print(df.isnull().sum() / len(df) * 100)  # % missing

# Unique values and counts
print(df['City'].value_counts())
print(df['City'].nunique())    # number of unique values

Output

Name     0
Age      1
Score    0
City     0
dtype: int64

Name     0.0
Age     20.0   ← 20% missing in Age column
Score    0.0
City     0.0
dtype: float64

Mumbai    2
Delhi     2
Pune      1
Name: City, dtype: int64
3

✂️

3. Selecting Rows & Columns

Access exactly the data you need — by label (loc) or by position (iloc)

loc vs iloc — the easiest way to remember

Method	Think	Example	Meaning
loc	Labels / names	df.loc[2, 'Score']	Row labelled 2, column named Score
iloc	Integer positions	df.iloc[2, 2]	3rd row, 3rd column

In the small table above, both examples reach Charlie's score: 92. The difference is how you point to the cell.

Python

# Select a single column → returns Series
print(df['Score'])

# Select multiple columns → returns DataFrame
print(df[['Name', 'Score']])

# loc — label based (use column names and index labels)
print(df.loc[0:2, 'Name':'Score'])  # rows 0-2, cols Name to Score

# iloc — position based (use integers like NumPy)
print(df.iloc[0:3, 1:3])           # rows 0-2, cols index 1-2

Output

0    88
1    75
2    92
3    68
4    95
Name: Score, dtype: int64

      Name  Score
0    Alice     88
1      Bob     75
2  Charlie     92
3    Diana     68
4      Eve     95

      Name  Age  Score
0    Alice   24     88
1      Bob   32     75
2  Charlie   28     92

    Age  Score
0  24.0     88
1  32.0     75
2  28.0     92

Python

# Boolean filtering — select rows meeting conditions
# Students who passed (score >= 70)
passed = df[df['Score'] >= 70]
print(passed)

# Multiple conditions: score >= 70 AND age < 30
young_passed = df[(df['Score'] >= 70) & (df['Age'] < 30)]
print(young_passed)

# From specific cities
delhi = df[df['City'].isin(['Delhi', 'Pune'])]
print(delhi)

Output

      Name   Age  Score    City
0    Alice  24.0     88   Delhi
1      Bob  32.0     75  Mumbai
2  Charlie  28.0     92   Delhi
4      Eve  22.0     95  Mumbai

      Name   Age  Score   City
0    Alice  24.0     88  Delhi
2  Charlie  28.0     92  Delhi
4      Eve  22.0     95  Mumbai

      Name   Age  Score   City
0    Alice  24.0     88  Delhi
2  Charlie  28.0     92  Delhi
3    Diana   NaN     68   Pune

Filtering is simply asking a question

df['Score'] >= 70 asks each row: “Is this student's score at least 70?”

Alice 88 ✓Bob 75 ✓Charlie 92 ✓Diana 68 ✗Eve 95 ✓

🧹

4. Cleaning Data — A Critical Step in ML

Garbage in, garbage out. Real datasets often have missing values, wrong types, and duplicates

A missing-value example with easy numbers

Diana's age is missing. The known ages are 24, 32, 28 and 22. If we choose mean imputation:

STEP 1 — ADD

24 + 32 + 28 + 22 = 106

STEP 2 — DIVIDE

106 ÷ 4 = 26.5

STEP 3 — FILL

Diana's missing Age → 26.5

Mean imputation is easy to understand, but it is not automatically the best method. The right choice depends on why values are missing and on the data distribution.

Python

# Handling missing values
print(df['Age'].isnull().sum())  # how many NaN in Age?

# Option 1: drop rows with missing values (use carefully)
df_dropped = df.dropna()

# Option 2: fill a numerical column with its mean
df['Age'] = df['Age'].fillna(df['Age'].mean())

# For a categorical column, the mode can sometimes be useful:
# df['City'] = df['City'].fillna(df['City'].mode()[0])

# For ordered/time-series data, forward fill may sometimes be suitable:
# df = df.ffill()

print(df.isnull().sum())

Output

1
Name     0
Age      0
Score    0
City     0
dtype: int64

Python

# Remove duplicates
df_clean = df.drop_duplicates()
print(f"Before: {len(df)}, After: {len(df_clean)}")

# Fix data types — models need numbers, not strings
df['Age'] = df['Age'].astype(int)
df['Score'] = df['Score'].astype(float)

# Rename columns
df.rename(columns={'Score': 'exam_score', 'Name': 'student_name'}, inplace=True)

# Drop columns not needed for ML
df.drop(columns=['student_name'], inplace=True)
print(df.dtypes)

Output

Before: 5, After: 5
Age            int64
exam_score    float64
City           object
dtype: object

⚙️

5. Creating New Features

Useful derived features can help a model learn patterns more effectively

Feature engineering means creating a more useful column from information you already have

Original

Score = 92

→ apply a simple rule →

New feature

Grade = A

Python

df = pd.DataFrame({
    'Name': ['Alice','Bob','Charlie','Diana'],
    'Age':  [24, 32, 28, 22],
    'Score':[88, 75, 92, 68],
    'City': ['Delhi','Mumbai','Delhi','Pune'],
})

# Create new column
df['Grade'] = df['Score'].apply(lambda x: 'A' if x >= 90 else ('B' if x >= 75 else 'C'))

# Bin age into groups
df['Age_Group'] = pd.cut(df['Age'], bins=[18,25,30,40],
                         labels=['18-25','26-30','31-40'])

# Extract info from strings
df['City_Short'] = df['City'].str[:3].str.upper()  # 'DEL', 'MUM', etc.

print(df[['Name','Score','Grade','Age_Group','City_Short']])

Output

      Name  Score Grade Age_Group City_Short
0    Alice     88     B     18-25        DEL
1      Bob     75     B     31-40        MUM
2  Charlie     92     A     26-30        DEL
3    Diana     68     C     18-25        PUN

📊

6. GroupBy — Aggregate & Summarise

Find patterns across groups — essential for EDA and feature engineering

GroupBy = Split → Calculate → Combine

For Delhi, Alice scored 88 and Charlie scored 92. To find Delhi's average:

88+92= 180→180 ÷ 2 = 90

Pandas performs this same idea for every city with groupby().

Python

# Average score by city
city_stats = df.groupby('City')['Score'].mean()
print(city_stats)

# Multiple aggregations at once
summary = df.groupby('City')['Score'].agg(['mean','min','max','count'])
print(summary)

# GroupBy on multiple columns
grade_city = df.groupby(['Grade','City'])['Score'].count()
print(grade_city)

Output

City
Delhi     90.0
Mumbai    75.0
Pune      68.0
Name: Score, dtype: float64

          mean  min  max  count
City
Delhi     90.0   88   92      2
Mumbai    75.0   75   75      1
Pune      68.0   68   68      1

Grade  City
A      Delhi     1
B      Delhi     1
       Mumbai    1
C      Pune      1
Name: Score, dtype: int64

🔗

7. Combining Tables with merge()

Real projects often store related information in separate tables

Students

1 Asha
2 Ravi

Scores

1 82
2 76

→

Merged by StudentID

1 Asha 82
2 Ravi 76

Python

students = pd.DataFrame({
    'StudentID': [1, 2],
    'Name': ['Asha', 'Ravi']
})

scores = pd.DataFrame({
    'StudentID': [1, 2],
    'Score': [82, 76]
})

combined = pd.merge(students, scores, on='StudentID')
print(combined)

Output

   StudentID  Name  Score
0          1  Asha     82
1          2  Ravi     76

The shared key StudentID tells Pandas which rows belong together. This is similar to joining tables in SQL.

🤖

8. Complete ML Preprocessing Pipeline

How to go from a raw CSV to model-ready features using pandas

The complete idea before the code

Raw CSV→Inspect→Clean→Encode categories→Choose X and y→Train / test split

Python

import pandas as pd
from sklearn.model_selection import train_test_split

# Step 1: Load data
df = pd.read_csv('titanic.csv')

# Step 2: Drop columns we do not want to use as features here
df = df.drop(columns=['PassengerId', 'Name', 'Ticket', 'Cabin'])

# Step 3: Fill missing values
df['Age'] = df['Age'].fillna(df['Age'].median())
df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])

# Step 4: One-hot encode nominal categories
# This avoids pretending that categories have a natural numeric order.
df = pd.get_dummies(
    df,
    columns=['Sex', 'Embarked'],
    drop_first=True,
    dtype=int
)

# Step 5: Separate features (X) and target (y)
X = df.drop(columns=['Survived'])
y = df['Survived']

# Step 6: Split directly — scikit-learn can work with DataFrames
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Training samples: {X_train.shape[0]}")
print(f"Test samples:     {X_test.shape[0]}")
print(f"Features:         {X_train.shape[1]}")

Output

Training samples: 712
Test samples:     179
Features:         8

Why one-hot encoding here? Columns such as Sex and Embarked are categories, not quantities. Giving them arbitrary numbers such as 0, 1 and 2 can incorrectly suggest an order or distance. One-hot encoding represents the categories more safely for many models.

🎯 Pandas Cheat Sheet — Most-Used ML Operations

Task	Code
Load CSV	pd.read_csv('file.csv')
Check shape	df.shape
Quick stats	df.describe()
Missing values	df.isnull().sum()
Fill missing	df['col'].fillna(df['col'].mean())
Drop duplicates	df.drop_duplicates()
Select rows	df[df['col'] > 50]
Group average	df.groupby('City')['Score'].mean()
Create feature	df['new'] = df['a'] + df['b']
To NumPy (when needed)	df.to_numpy()
Merge tables	pd.merge(left, right, on='id')
Rename column	df.rename(columns={'old':'new'})
Drop column	df.drop(columns=['col'])

Common Pandas Questions for ML Beginners

What is the difference between a Series and a DataFrame?

A Series is one labelled column of values. A DataFrame is a table made of multiple labelled columns.

What is the difference between loc and iloc?

loc selects using labels or names. iloc selects using integer positions.

Why is Pandas useful before machine learning?

Real data usually needs inspection, cleaning, filtering, combining and feature preparation before a model can learn from it.

Does scikit-learn require NumPy arrays?

No. Many scikit-learn estimators can accept Pandas DataFrames and Series directly. Converting to NumPy is useful in some workflows, but it is not always required.

Should I always fill missing numerical values with the mean?

No. Mean imputation is one simple option. The right method depends on the data, the amount and pattern of missingness, and the model you plan to use.

Continue learning

Pandas is mainly about preparing tabular data. These lessons build directly on the skills used here:

NumPy Essentials Exploratory Data Analysis Handling Missing Data Feature Engineering Scikit-learn Essentials

Pandas Essentials for ML

Pandas for Machine Learning

Pandas in Simple Words

1. Creating & Loading DataFrames

2. Exploring Your Dataset — Always Do This First

3. Selecting Rows & Columns

loc vs iloc — the easiest way to remember

4. Cleaning Data — A Critical Step in ML

A missing-value example with easy numbers

5. Creating New Features

6. GroupBy — Aggregate & Summarise

GroupBy = Split → Calculate → Combine

7. Combining Tables with merge()

8. Complete ML Preprocessing Pipeline

Common Pandas Questions for ML Beginners

Continue learning

Apply this concept in a real hands-on project

Finished "Pandas Essentials for ML"?

Was this tutorial helpful?

Summary / Key Takeaways

Knowledge Check

Knowledge Check Review & Explanations

Continue Learning

Related topics

Free Download: ML Interview Cheatsheet

Optional Learning Resources

Hands-On Machine Learning