Naive Bayes

Naive Bayes Classifier

Naive Bayes is a probabilistic classifier based on applying Bayes’ Theorem with a strong (naive) conditional-independence assumption between features given the class.

Naive Bayes in Simple Words

Naive Bayes looks at clues, checks how often those clues appeared with each class in the past, and chooses the class with the strongest probability score. A simple example is email spam detection.

What is Naive Bayes?

Naive Bayes is a supervised machine learning algorithm used mainly for classification problems. It predicts the probability that a data point belongs to a particular class based on Bayes' Theorem combined with a Feature Independence Assumption.

Common Uses:

• Spam Detection: Filtering out junk emails.
• Sentiment Analysis: Understanding positive/negative product reviews.
• Document Classification: Organizing news articles into categories.
• Medical Data Classification: Supporting carefully validated research or decision-support tasks.

Simple Example: Spam Detection

Suppose an email contains words like: FREE, WINNER, MONEY, CLICK

Why is it Called "Naive"?

The “naive” assumption is more specific than saying that features are always unrelated. Naive Bayes treats the features as conditionally independent once the class is known.

Bayes’ Theorem Formulas

Naive Bayes is built directly on Bayes’ Theorem. Bayes’ Theorem helps calculate conditional probability — the probability of an event given another event has occurred.

A Tiny Bayes Calculation

Suppose past email data tells us:

Real-Life Analogy: A common probability example is medical testing: if a condition is rare but a test is positive, Bayes’ theorem updates the probability after observing that result. In real healthcare, diagnosis requires validated clinical evidence and cannot be reduced to one toy calculation.

Worked Numerical Example

Let's predict whether a day is suitable for playing golf based on the weather outlook.

Types of Naive Bayes

Naive Bayes is not a single algorithm — it is a family of classifiers, each designed for a different statistical distribution of features. The appropriate variant depends mainly on how the input features are represented and what probability distribution the model assumes for them.

from sklearn.naive_bayes import GaussianNB
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Load dataset (continuous features: sepal/petal length & width)
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train Gaussian NB — automatically estimates μ and σ per class per feature
model = GaussianNB()
model.fit(X_train, y_train)

# Class-conditional means learned during training:
print(model.theta_)   # shape: [n_classes, n_features] → mean per class
print(model.var_)     # shape: [n_classes, n_features] → variance per class

predictions = model.predict(X_test)
print(f"Accuracy: {accuracy_score(y_test, predictions):.2%}")
# Output with this split: Accuracy: 100.00%

from sklearn.naive_bayes import BernoulliNB
from sklearn.feature_extraction.text import CountVectorizer
import numpy as np

emails = [
    "win free money now",      # spam
    "click here for prize",    # spam
    "meeting tomorrow at 10",  # ham
    "quarterly report attached", # ham
]
labels = [1, 1, 0, 0]  # 1=spam, 0=ham

# binary=True converts word counts to 0/1 presence flags
vectorizer = CountVectorizer(binary=True)
X = vectorizer.fit_transform(emails)
# Each row: [0,1,0,0,1,...] — word present (1) or absent (0)

model = BernoulliNB()
model.fit(X, labels)

new_email = vectorizer.transform(["free prize click"])
print(model.predict(new_email))         # [1] → classified as SPAM
print(np.round(model.predict_proba(new_email), 3))
# [[0.053 0.947]] → about 94.7% spam in this tiny example

from sklearn.naive_bayes import ComplementNB
from sklearn.feature_extraction.text import TfidfVectorizer

# Complement NB is a drop-in replacement for MultinomialNB
# Best for: imbalanced text classification tasks
vectorizer = TfidfVectorizer()
X_train_tfidf = vectorizer.fit_transform(X_train_text)
X_test_tfidf  = vectorizer.transform(X_test_text)

model = ComplementNB(alpha=1.0)  # alpha = Laplace smoothing
model.fit(X_train_tfidf, y_train)
print(f"Accuracy: {model.score(X_test_tfidf, y_test):.2%}")

The Zero Probability Problem

In count-based variants such as Multinomial Naive Bayes, a word that never appeared in a class can receive an estimated probability of 0. Multiplying by zero can wipe out that class score. Additive smoothing prevents this problem.

Python Implementations

from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer

texts = ["win free money", "hello friend", "claim prize free"]
y = ["spam", "ham", "spam"]

# Convert text to frequency counts
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)

# Train Model
model = MultinomialNB()
model.fit(X, y)

# Predict a new email
test_email = ["free money now"]
test_vector = vectorizer.transform(test_email)
prediction = model.predict(test_vector)

print(f"Prediction for '{test_email[0]}': {prediction[0]}")

Advantages vs Disadvantages

Common Questions

Final Summary

Naive Bayes is a classic probabilistic classification method built on Bayes' theorem. It combines a class prior with feature likelihoods to score possible classes for a new observation.

Its conditional-independence assumption is deliberately simple, yet Naive Bayes can still be a strong and very fast baseline in suitable problems, especially text classification.