Cost Functions
A Machine Learning model often needs a numerical way to answer one simple question: “How bad are my current predictions?” A loss or cost function provides that signal.
Cost Functions in Simple Words
STEP 1
Predict
Model makes a prediction
STEP 2
Compare
Compare prediction with the target
STEP 3
Measure Loss
Turn the mistake into a number
STEP 4
Improve
Training tries to reduce the objective
Real-Life Analogy — Archery
Imagine learning archery. After each shot you need feedback about how far the arrow landed from the target. A loss function plays a similar role: it converts prediction quality into a number that an optimization procedure can use when adjusting a trainable model.
Important terminology
Books and libraries do not use the words loss, cost, and objectivein exactly the same way. A useful convention is: a loss measures one example, a cost aggregates losses over data, and an objective may additionally contain regularization. Treat this as a helpful vocabulary guide, not a universal naming law.
Loss
Often refers to the penalty for one training example.
Actual 70, Predicted 66 → squared loss = 4² = 16
Cost
Often means an average or sum of losses over many examples.
Cost = average(loss₁, loss₂, ...)
Objective
What training actually optimizes; it may include a penalty.
Objective = data loss + regularization
Not every Machine Learning algorithm trains by differentiating a cost function. Gradient-based models do, while Decision Trees choose splits using criteria such as impurity reduction, and KNN mainly stores examples instead of learning parameters through Gradient Descent. The broad idea is still the same: the algorithm needs a rule for deciding what counts as a better model or decision.
From Prediction Error to a Useful Loss
For a regression prediction, a signed residual can be written as:
Residual = y - ŷ
- y = actual target
- ŷ = predicted target
Why Simply Adding Signed Errors Is Usually Not Enough
Positive and negative residuals can cancel each other even when every prediction is wrong.
| Actual | Predicted | Residual |
|---|---|---|
| 50 | 55 | -5 |
| 60 | 55 | +5 |
| Sum | 0 | |
A sum of zero clearly does not mean the predictions are perfect. Regression losses therefore commonly use operations such as squaring or absolute value. Classification losses use other constructions, such as logarithmic penalties for probabilities.
Regression Loss Functions
Regression predicts continuous values such as house price, temperature, demand, or delivery time. Different losses place different importance on small and large errors.
1. Mean Squared Error (MSE)
MSE = (1/n) Σ (yᵢ - ŷᵢ)²
MSE squares each error and then averages. Squaring makes larger errors contribute disproportionately more to the total loss.
MSE Worked Example — House Prices (₹000s)
Actual: [100, 150, 200, 250] | Predicted: [90, 140, 220, 230]
- Residuals:
10, 10, -20, 20 - Square them:
100, 100, 400, 400 - Add:
100 + 100 + 400 + 400 = 1000 - Average:
1000 / 4 = 250
MSE = 250
What to remember
Because errors are squared, an unusually large error can have a strong influence on MSE. That can be useful when large misses are especially costly, but it also makes MSE more sensitive to outliers than MAE.
2. Mean Absolute Error (MAE)
MAE = (1/n) Σ |yᵢ - ŷᵢ|
MAE Worked Example — Same Predictions
- Residuals:
10, 10, -20, 20 - Absolute errors:
10, 10, 20, 20 - Add:
10 + 10 + 20 + 20 = 60 - Average:
60 / 4 = 15
MAE = 15
3. Root Mean Squared Error (RMSE)
RMSE is simply the square root of MSE. This returns the error measure to the original target units.
MSE = 250
RMSE = √250
RMSE ≈ 15.81
4. Huber Loss
Huber-style loss behaves quadratically for smaller errors and more linearly for sufficiently large errors. This can reduce the influence of outliers compared with pure squared loss while keeping a smooth region near zero. The transition point is controlled by a threshold parameter.
MSE
Large errors receive rapidly increasing weight because they are squared.
MAE
Penalty grows linearly with absolute error.
Huber
Quadratic near zero, more linear for larger errors.
Classification Loss Functions
Classification often works with class probabilities or decision scores. For probabilistic classifiers, Cross-Entropy / Log Loss is a natural choice because it rewards assigning high probability to the true class and strongly penalizes confident probability mistakes.
A useful nuance
MSE is not universally “forbidden” for classification. It can be optimized in some model formulations, but Cross-Entropy is generally a better-matched objective for probability-based classification and is standard for Logistic Regression-style models.
Binary Cross-Entropy (Log Loss)
Loss = -[ y log(p) + (1-y) log(1-p) ]
Correct and confident
Actual y = 1
Predicted p = 0.95
Loss = -log(0.95)
≈ 0.051
Confident and wrong
Actual y = 1
Predicted p = 0.02
Loss = -log(0.02)
≈ 3.912
Notice the important idea: both examples have the same true class, but the model receives a much larger loss when it assigns only 2% probability to that true class.
Multiclass Cross-Entropy
With several classes, the model assigns a probability to each class and the loss focuses on the probability assigned to the true class.
Suppose the true class is Cat and the model predicts:
Cat = 0.70, Dog = 0.20, Horse = 0.10
Loss = -log(0.70)
≈ 0.357
Hinge Loss, Tree Criteria, and Reward
Hinge Loss — Margin-Based Classification
Hinge loss is closely associated with linear Support Vector Machine objectives. It penalizes a point not only when it is misclassified, but also when it lies on the correct side with insufficient margin.
Loss = max(0, 1 - y · f(x))
For this binary form, y is +1 or -1 and f(x) is the decision score.
Worked Example
- y = 1, f(x) = 1.5:
max(0, 1 - 1.5) = 0 - The point is correctly classified and beyond the unit-margin threshold.
- y = 1, f(x) = 0.2:
max(0, 1 - 0.2) = 0.8 - The sign is correct, but the margin is too small.
Gini Impurity / Entropy — Decision Tree Split Criteria
Decision Trees are different from gradient-based models. They evaluate candidate splits using criteria such as Gini impurity or entropy. It is clearer to call these split criteria / impurity measures rather than pretending they are the same kind of training loss used by Logistic Regression.
Worked Example — Gini Impurity
Gini = 1 - Σ pᵢ²
A node contains 4 Apples and 6 Oranges.
p(Apple) = 4/10 = 0.4p(Orange) = 6/10 = 0.6Gini = 1 - (0.4² + 0.6²)= 1 - (0.16 + 0.36) = 0.48
A tree compares candidate splits by how much they reduce impurity after accounting for the child-node sizes.
Reward Function — Reinforcement Learning
Reinforcement Learning is another important distinction. The environment gives a reward signal, and the agent typically aims to maximize expected cumulative reward. Some mathematical formulations minimize a negative objective, but a reward function should not automatically be called a supervised-learning cost function.
Simple Game Example
- Collect useful item → Reward +10
- Waste a move → Reward -2
- Lose the game → Reward -100
The learning objective concerns the expected return over time, not merely minimizing the immediate negative reward of one action.
Training Objective vs Evaluation Metric
The quantity optimized during training does not have to be the same quantity reported to stakeholders.
Training objective
Used internally to fit model parameters.
- Log loss for Logistic Regression
- Squared error for many regression models
- Loss + regularization for Ridge/Lasso-style objectives
Evaluation metric
Used to judge whether a trained model is useful.
- Accuracy, precision, recall, F1
- MAE, RMSE, R²
- Business-specific cost or service-level metrics
Example
A fraud classifier might be trained using log loss but evaluated using recall, precision, PR-AUC, and the financial cost of missed fraud. Optimizing one differentiable training objective does not mean it is the only metric that matters.
How Loss Connects to Optimization
When the objective is differentiable, an optimizer such as Gradient Descent can use derivatives to decide how parameters should change. In neural networks, gradients are computed efficiently through backpropagation.
Gradient Descent Analogy
Imagine standing on a hill and repeatedly stepping in a direction that reduces altitude. The terrain is the objective surface, and the gradient describes the local slope. This analogy is useful for gradient-based models, but not every ML algorithm trains by walking down such a surface.
Python Example
Scikit-learn's metric functions let us calculate several common losses directly.
from sklearn.metrics import (
mean_squared_error,
mean_absolute_error,
log_loss,
)
actual_reg = [100, 150, 200, 250]
predicted_reg = [90, 140, 220, 230]
mse = mean_squared_error(actual_reg, predicted_reg)
mae = mean_absolute_error(actual_reg, predicted_reg)
rmse = mse ** 0.5
print(f"MSE: {mse:.1f}")
print(f"MAE: {mae:.1f}")
print(f"RMSE: {rmse:.2f}")
actual_clf = [1, 0, 1]
predicted_prob = [0.95, 0.10, 0.02]
loss = log_loss(actual_clf, predicted_prob)
print(f"Log Loss: {loss:.4f}")Expected output
MSE: 250.0 MAE: 15.0 RMSE: 15.81 Log Loss: 1.3562
The combined log loss is relatively large because the third example assigns only 0.02 probability to a class that is actually positive.
Quick Comparison
| Measure | Typical use | Main idea |
|---|---|---|
| MSE | Regression | Squares errors; large errors receive more weight |
| MAE | Regression | Uses absolute error; linear penalty |
| Huber | Robust regression | Quadratic for smaller errors, more linear for larger errors |
| Cross-Entropy | Probabilistic classification | Penalizes low probability assigned to the true class |
| Hinge | Margin-based classification | Penalizes margin violations |
| Gini / Entropy | Decision Trees | Evaluate candidate split quality |
| Reward | Reinforcement Learning | Defines desirable outcomes over sequential interaction |
Common Mistakes
Calling every evaluation metric a training cost function.
Assuming lower training loss always means better performance on unseen data.
Choosing MSE or MAE only because one number looks smaller—their numerical scales are different.
Treating Decision Tree impurity and Reinforcement Learning reward as if they were identical to supervised differentiable losses.
Optimizing a convenient ML loss while ignoring the real business cost of different mistakes.
Quick Recap
Why not simply add signed regression errors?
Positive and negative residuals can cancel even when the individual predictions are wrong.
Why does MSE react strongly to large errors?
Because each error is squared before averaging.
Is the training loss always the final evaluation metric?
No. A model may optimize one objective and be judged using different validation, test, or business metrics.
Most Important Insight
A loss function defines what kind of mistake the learning procedure should care about. Choosing it is therefore not just a mathematical detail: it determines the behavior the model is encouraged to learn.
Gradient Descent →
See how differentiable objectives guide parameter updates.
Logistic Regression →
Connect probabilities with log loss.
Hyperparameter Tuning →
Next: choose model settings using validation evidence.