ROC-AUC

ROC-AUC in Machine Learning

A classifier often produces a score for each example: for instance, a fraud probability of 0.82 or a decision score from an SVM. A threshold then turns that score into a final class such as Fraud / Not Fraud.

What Is the ROC Curve?

ROC stands for Receiver Operating Characteristic. For binary classification, the ROC curve plots the True Positive Rate (TPR) against the False Positive Rate (FPR) as the decision threshold changes.

A tiny score example

Why changing the threshold matters

Lowering the threshold caught the remaining positive example, but it also created a false alarm. The ROC curve records this trade-off over many thresholds.

Understanding the ROC Axes

One simple ROC picture

Curves that reach toward the top-left generally indicate stronger ranking performance: high TPR can be achieved while keeping FPR relatively low. The diagonal represents random ranking in the usual binary ROC interpretation.

What Is AUC?

AUC means Area Under the ROC Curve. It compresses the ROC curve into one number that summarizes how well the model's scores rank positive examples above negative examples across thresholds.

How ROC Points Are Created

Start with the score table from earlier. At each threshold, convert scores into positive/negative predictions, count TP/FP/FN/TN, and calculate TPR and FPR.

Repeating this process across score thresholds creates the sequence of ROC points. AUC summarizes the resulting curve.

Comparing Models with ROC-AUC

ROC-AUC is useful when you care about how well models rank positives above negatives across many possible thresholds. It can reveal differences that a single fixed-threshold accuracy score hides.

ROC-AUC Does Not Choose Your Final Threshold

ROC-AUC evaluates ranking across thresholds. Deployment still needs a particular decision threshold, and that threshold should reflect the real problem.

For example, if investigators can review only a limited number of alerts each day, a threshold that produces an impractical number of false alarms may be unusable even when the overall ROC-AUC is strong.

Python Implementation

This example uses a leakage-safe Pipeline, an explicit stratified train/test split, continuous positive-class probabilities, and Scikit-learn's ROC utilities.

import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, roc_auc_score, RocCurveDisplay

# 1. Load educational binary-classification data
X, y = load_breast_cancer(return_X_y=True)

# 2. Keep a final test set
X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.30,
    random_state=42,
    stratify=y
)

# 3. Scale using training data inside the Pipeline, then fit
model = make_pipeline(
    StandardScaler(),
    LogisticRegression(max_iter=2000)
)
model.fit(X_train, y_train)

# 4. Use continuous scores for the positive class
#    (decision_function scores can also be used by many estimators)
y_score = model.predict_proba(X_test)[:, 1]

# 5. Compute ROC points and ROC-AUC
fpr, tpr, thresholds = roc_curve(y_test, y_score)
auc_score = roc_auc_score(y_test, y_score)

print(f"AUC Score: {auc_score:.3f}")
print(f"ROC points: {len(fpr)}")

# 6. Plot from the same predictions
RocCurveDisplay.from_predictions(y_test, y_score)
plt.plot([0, 1], [0, 1], "--", label="Random ranking")
plt.legend()
plt.show()

AUC Score: 0.998
ROC points: 6

ROC Curve vs Precision-Recall Curve

ROC and Precision-Recall curves answer related but different questions. When the positive class is very rare, a Precision-Recall curve can often be especially informative because precision directly reflects how many predicted positives are actually positive.

What About Multiclass Problems?

The basic ROC curve is easiest to understand for binary classification. ROC-AUC can also be extended to multiclass and multilabel problems using strategies such as one-vs-rest or one-vs-one and then combining class-level scores with averaging choices.

For a first lesson, master the binary case first. The key idea remains the same: evaluate ranking scores rather than relying only on one hard threshold.

Common Mistakes

Quick Recap