Logistic Regression
Logistic Regression is a classification algorithm used to predict categorical outcomes.
It is used to predict categories such as:
- Spam or Not Spam
- Disease or No Disease
- Fraud or Legitimate
- Pass or Fail
- Customer Will Buy or Not Buy
Logistic Regression is a widely used baseline because it is relatively simple, fast to train, and often interpretable. It is used in areas such as healthcare, finance, marketing, and cybersecurity.
Logistic Regression in Simple Words
Think of Logistic Regression as a model that first gives a new example a score, converts that score into an estimated probability, and then compares that probability with a threshold to choose a class.
| Study Hours | Attendance | Known Result |
|---|---|---|
| 1 | 55% | Fail |
| 2 | 65% | Fail |
| 4 | 82% | Pass |
| 5 | 90% | Pass |
A new student arrives.
Suppose the trained model estimates:
Probability of Pass = 0.82
If our chosen threshold is 0.50, then 0.82 ≥ 0.50, so the model predicts Pass. Later in this lesson you will see where that probability comes from.
What is Logistic Regression?
Logistic Regression is a supervised machine learning algorithm used for predicting probabilities and classifying data into categories.
Simple Beginner Definition
Bank Loan (Finance)
Suppose a bank wants to predict whether a customer will repay a loan. Input features are Salary, Age, Credit Score, and Loan Amount.
The model learns patterns to predict the probability of repayment.
Diabetes (Healthcare)
Suppose a hospital wants to predict whether a patient has diabetes. Input features are Age, BMI, Blood Pressure, and Glucose Level.
The model learns patterns from patient records and predicts new cases.
Where Logistic Regression Fits
Why Logistic Regression is Needed
Linear Regression works well for predicting numbers but fails for classification. Suppose we use Linear Regression for spam detection.
Linear Regression Problem
Possible continuous outputs:
- -2.3
- 4.7
- 12.5
These values do not represent probabilities.
Logistic Regression Solution
Uses the Sigmoid Function. We need outputs strictly between:
0 and 1
Produces model-estimated probabilities that can be converted into class decisions.
| Feature | Regression | Classification |
|---|---|---|
| Output | Continuous number | Category |
| Example | House price | Spam/Not Spam |
| Algorithms | Linear Regression | Logistic Regression |
Logistic Regression Workflow
Types of Logistic Regression
1. Binary
Used when there are only two classes.
- YES / NO
- TRUE / FALSE
- SPAM / NOT SPAM
2. Multinomial
Used when there are multiple categories.
- Cat
- Dog
- Horse
- Tiger
3. Ordinal Logistic Regression
A related specialized extension for ordered outcomes. Standard logistic-regression APIs do not automatically use the order.
- Low
- Medium
- High
Core Idea of Logistic Regression
Logistic Regression first calculates a linear equation similar to Linear Regression.
Linear Equation
z = b₀ + b₁x₁ + b₂x₂ + ... + bₙxₙ
where z = linear output, b₀ = intercept, b₁, b₂ = coefficients, x₁, x₂ = features
Worked Example
Suppose b₀ = -3, b₁ = 1, x = 4
z = -3 + 1(4)
z = -3 + 4
z = 1
But 1 is still a score, not yet a probability.
So Logistic Regression applies the Sigmoid Function.
The Sigmoid Function
The sigmoid function converts linear values into probabilities strictly between 0 and 1.
Sigmoid Formula
σ(z) = 1 / (1 + e⁻ᶻ)
How it limits output to 0..1:
- e is Euler's number (≈ 2.718).
- When z is highly positive, e⁻ᶻ becomes very small, making the denominator ≈ 1, so the output approaches 1.
- When z is highly negative, e⁻ᶻ becomes huge, making the denominator huge, so the output approaches 0.
- When z = 0, e⁰ = 1, making the output 1 / (1+1) = 0.5.
Why Sigmoid is Used
- Converts numbers into probabilities: Ensures outputs are strictly between 0 and 1.
- Smooth and differentiable: This makes the model convenient to optimize with gradient- or Newton-style numerical methods.
- Monotonic: A larger linear score always maps to a larger estimated probability. The decision boundary remains linear in the features unless we first create nonlinear features.
Three Easy Sigmoid Values to Remember
| z | σ(z) | Meaning |
|---|---|---|
| -2 | ≈ 0.119 | Closer to Class 0 |
| 0 | 0.500 | Middle point |
| 2 | ≈ 0.881 | Closer to Class 1 |
As z increases, the sigmoid probability increases smoothly. You do not need to memorize the whole curve; understand the direction.
Sigmoid & Decision Threshold Explorer
See how changing Weights (w), Bias (b), and Decision Threshold (T) converts continuous linear scores into probabilities and binary classes.
Confusion Matrix on 10 Sample Patients
Acc: 100%Worked Example of Sigmoid Function
Suppose: z = 2
σ(2) = 1 / (1 + e⁻²)
(Approx: e⁻² ≈ 0.135)
Step 1: e⁻² ≈ 0.1353
Step 2: 1 + 0.1353 = 1.1353
Step 3: 1 / 1.1353 ≈ 0.881
σ(2) ≈ 0.881
Meaning: about 88.1% model-estimated probability of Class 1
Probability Threshold & Decision Boundary
A predicted probability can be converted into a class label using a chosen threshold. A threshold of 0.5 is a common starting point for binary classification, but it is not automatically the best threshold. The decision boundary is where the model switches from one class decision to the other.
Threshold Logic
Same probability, different threshold
Threshold = 0.50 → Class 1
Threshold = 0.80 → Class 0
Simple Decision Boundary
In the original feature space, standard Logistic Regression creates a linear boundary. Nonlinear feature engineering can make the resulting boundary more flexible.
Cost Function & Gradient Descent
Log Loss Formula
Unlike Linear Regression, Logistic Regression uses Log Loss.
- y: Actual label (0 or 1)
- ŷ: Predicted probability of Class 1
- m: Number of samples
One-sample example
Actual label: y = 1
Predicted probability: ŷ = 0.8
Loss = -log(0.8)
Loss ≈ 0.223
A confident correct prediction gets a small loss. If the model predicted 0.1 for this positive example, the loss would be about 2.303.
Why Log Loss Fits Logistic Regression
Log Loss matches the probabilistic model used by binary Logistic Regression. It strongly penalizes confident wrong predictions.
- Uses the predicted probability directly
- Rewards confident correct predictions
- Penalizes confident wrong predictions heavily
MSE can be computed for probabilities, but it is not the standard training objective for ordinary Logistic Regression.
Gradient Descent Intuition
One way to understand optimization is to repeatedly move the coefficients in a direction that reduces Log Loss. Practical libraries may use gradient-based, Newton-style, or coordinate-based solvers depending on the settings.
θ = coefficients, α = learning rate, J = cost function
Learning Workflow
Odds, Logit Function & Scaling
Odds Definition
Logistic Regression uses odds internally.
Worked Example:
Suppose P = 0.8
Odds = 0.8 / (1 - 0.8) = 0.8 / 0.2
Odds = 4
Meaning: the positive outcome has odds of 4 to 1 against the negative outcome.
Logit Function
The logit converts a probability into log-odds. Logistic Regression models these log-odds as a linear combination of the features.
Feature Scaling
Feature scaling is often helpful when numerical features have very different units. It can make optimization more stable for some solvers and makes regularization treat coefficient scales more fairly.
Example:
X = 100, Mean(μ) = 80, StdDev(σ) = 10
Step 1: 100 - 80 = 20
Step 2: 20 / 10 = 2
X_scaled = 2
Evaluation Metrics
Confusion Matrix
| Predicted | ||
|---|---|---|
| YES | NO | |
| Actual YES | TP | FN |
| Actual NO | FP | TN |
Core Formulae
- AccuracyAcc = (TP+TN)/(TP+TN+FP+FN)
- PrecisionPrec = TP / (TP+FP)
- RecallRecall = TP / (TP+FN)
- F1 ScoreF1 = 2*(Prec*Rec)/(Prec+Rec)
Metric Example with Small Numbers
Suppose TP = 8, TN = 7, FP = 1, FN = 2.
ROC Curve
ROC Curve measures classification performance across various thresholds.
It plots True Positive Rate vs False Positive Rate.
Comparisons & Applications
Logistic vs Linear Regression
| Output | Probability | Continuous |
|---|---|---|
| Function | Sigmoid | Linear |
| Cost Fx | Log Loss | MSE |
Logistic vs Decision Tree
| Boundary | Linear in chosen feature space | Piecewise, axis-aligned splits |
|---|---|---|
| Complexity | Usually lower | Depends on tree depth |
| Overfit Risk | Depends on regularization | Can be high for deep trees |
Industry Applications
Healthcare
- Disease Prediction
- Cancer Detection
- Medical Risk Analysis
Finance
- Credit Scoring
- Fraud Detection
- Loan Approval
Marketing
- Customer Conversion
- Churn Prediction
- Ad Analytics
Python Implementation (Practical Guide)
Here is a complete, beginner-friendly step-by-step implementation of Logistic Regression using Python and Scikit-Learn's Breast Cancer Wisconsin dataset. It is a naturally binary classification dataset, so we do not need to manufacture labels from a regression target.
Machine Learning Pipeline Workflow
Python ML Libraries
• StandardScaler
• Train/Test Split
• Metrics
Natural Binary Labels
This dataset already contains two target classes, so the example stays a genuine classification problem.
Educational dataset only; this tutorial is not a clinical diagnostic tool.
Interactive: Train Logistic Regression in Python
Adjust test set split size and max iterations, then execute scikit-learn's LogisticRegression model live.
Proportion of the dataset held out for unbiased evaluation.
Maximum solver steps for lbfgs optimization to reach convergence.
| 1 | from sklearn.datasets import load_breast_cancer |
| 2 | from sklearn.model_selection import train_test_split |
| 3 | from sklearn.preprocessing import StandardScaler |
| 4 | from sklearn.linear_model import LogisticRegression |
| 5 | from sklearn.metrics import accuracy_score, confusion_matrix, classification_report, roc_auc_score |
| 6 | |
| 7 | # Step 1: Load Dataset |
| 8 | data = load_breast_cancer() |
| 9 | X, y = data.data, data.target |
| 10 | |
| 11 | # Step 2: Split Before Scaling |
| 12 | test_size = 0.2 |
| 13 | X_train, X_test, y_train, y_test = train_test_split( |
| 14 | X, y, test_size=0.2 |
| 15 | ) |
| 16 | |
| 17 | # Step 3: Standardize Features |
| 18 | scaler = StandardScaler() |
| 19 | X_train_scaled = scaler.fit_transform(X_train) |
| 20 | X_test_scaled = scaler.transform(X_test) |
| 21 | |
| 22 | # Step 4: Train Logistic Regression |
| 23 | max_iter = 1000 |
| 24 | model = LogisticRegression(max_iter=1000 |
| 25 | model.fit(X_train_scaled, y_train) |
| 26 | |
| 27 | # Step 5: Predict & Evaluate |
| 28 | y_pred = model.predict(X_test_scaled) |
| 29 | y_prob = model.predict_proba(X_test_scaled)[:, 1] |
| 30 | |
| 31 | acc = accuracy_score(y_test, y_pred) |
| 32 | auc = roc_auc_score(y_test, y_prob) |
| 33 | |
| 34 | print(f"Accuracy: {acc:.4f}") |
| 35 | print("Confusion Matrix:") |
| 36 | print(confusion_matrix(y_test, y_pred)) |
| 37 | print(f"ROC AUC Score: {auc:.4f}") |
| 38 | print("\nClassification Report:") |
| 39 | print(classification_report(y_test, y_pred, target_names=data.target_names)) |
Expected Output
[[41 1]
[ 1 71]]
malignant precision 0.98 recall 0.98 f1-score 0.98
benign precision 0.99 recall 0.99 f1-score 0.99
ROC AUC: 0.9953703703703703
Example output from the fixed split shown above. Small numerical differences can occur across library versions. A strong score on this teaching dataset should not be interpreted as proof of clinical readiness.
Best Practices & Common Mistakes
Common Mistakes
- Ignoring feature scaling: Can slow or destabilize optimization when feature scales differ greatly.
- Using wrong threshold: Can create the wrong trade-off between false positives and false negatives.
- Imbalanced data: Results in misleading aggregate accuracy scores.
Best Practices
- Scale numerical features when feature scales differ substantially or the chosen solver/regularization benefits from it.
- Use cross-validation.
- Tune threshold carefully based on business context.
- Handle class imbalance thoughtfully; consider class weights, resampling, and suitable metrics rather than applying one technique automatically.
- Evaluate multiple metrics (F1, Precision, Recall).
- Use Regularization (L1/L2) if needed.
Regularization in Scikit-Learn Logistic Regression
Scikit-Learn's LogisticRegression applies regularization by default. The parameter Ccontrols its strength inversely.
The available L1, L2, or Elastic-Net choices depend on the solver. Choose settings with validation rather than assuming one value is always best.
Common Questions
Because it models the log-odds as a regression-like linear equation, then converts the result to a probability and class decision.
No. The threshold should reflect validation results and the real cost of false positives versus false negatives.
Yes. Multinomial logistic regression can handle multiple unordered classes. Ordered outcomes require specialized ordinal models.
Continue with Decision Trees, or revisit Confusion Matrix and ROC-AUC for evaluation.
Final Summary
Logistic Regression is a classification algorithm in Machine Learning. It works by taking a linear combination of inputs, passing them through a Sigmoid Function, outputting a probability, and assigning a class based on a threshold.
Its major strengths include simplicity, computational efficiency, and coefficient-based interpretability. It can provide useful estimated probabilities, but those probabilities should still be evaluated for calibration and used with a threshold that fits the problem. It is a strong baseline for many binary and multiclass classification tasks.
Most Important Insight to Remember:
"The key idea is simple: Logistic Regression turns a linear score into an estimated probability, then uses a decision rule to choose a class."