What is the difference between in-sample fit and out-of-sample forecast accuracy?

In-sample measures how well the model fits historical training data; out-of-sample measures performance on future data never seen. A model can have excellent in-sample fit (memorising noise) and terrible out-of-sample performance — A complex ARIMA might fit every historical wiggle perfectly but produce poor forecasts. Always evaluate on a held-out future period the model never saw.

What is the naive forecast and why is it a critical baseline?

The naive forecast predicts the next value equals the last observed value (stationary) or the same period last year (seasonal). If your complex model doesn't beat this, it has learned nothing useful from the data structure — The seasonal naive model is surprisingly hard to beat for many business series. If your model beats seasonal naive, it is demonstrably adding value.

Why is RMSE more commonly reported than MAE in business forecasting?

RMSE penalises large errors more heavily — in business, being very wrong once (stockout, overproduction) is disproportionately more costly than being slightly wrong many times. RMSE's penalty structure matches this cost — A supply chain off by 100 units 100 times vs off by 10,000 units once — the second situation is far more costly. RMSE makes the single large error much more visible.

What is a prediction interval and how does it differ from a confidence interval?

A prediction interval gives a range for an individual future observation (accounting for model uncertainty AND inherent series variability). A confidence interval gives a range for the mean forecast (model uncertainty only). For operational planning, prediction intervals are what you need — For inventory planning, you need the range of likely demand for a specific future day — that is a prediction interval. The wider prediction interval includes both model uncertainty and irreducible variability.

Time Series Forecasting Basics

Forecasting Basics — Complete Guide

Learn the forecasting workflow: understand time patterns, define the horizon, build simple baselines, evaluate on future periods, and only then add model complexity.

Time series forecasting means using information available up to a point in time to estimate values that come later. The key difference from ordinary tabular prediction is that time order matters. A forecasting system must behave as if the future has not happened yet.

Past data

Collect observations available up to the forecast origin.

Find structure

Look for level, trend, seasonality, cycles, events and useful predictors.

Forecast

Use a baseline or model to estimate one or more future periods.

Evaluate later

Compare forecasts with periods that were not used to fit or tune the model.

Forecasting is not the same as every prediction problem that mentions the future.

Predicting next month's sales or tomorrow's electricity demand is forecasting because the target is ordered over time. Predicting whether a customer will churn can be a classification problem instead. The question is not only when the outcome happens, but whether the target itself is a time-indexed quantity that must be forecast over a horizon.

Where Forecasting Is Used

Business

Retail sales by day or month
Revenue and cash-flow planning
Product demand and inventory
Website traffic and orders

Operations

Energy demand
Call-centre volume
Server load
Staffing requirements

Environment & Finance

Temperature or rainfall series
Power generation
Transaction volume
Financial time-series quantities

Two Words You Must Know: Forecast Origin and Horizon

The forecast origin is the last point at which you are allowed to use observed information. The forecast horizon tells you how far ahead you need to predict.

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug

Sep

Observed pastForecast origin: end of June3-step horizon: Jul–Sep

One-step-ahead forecasting

Predict the next period, observe it when it arrives, then forecast the following period. New actual values may become available between forecasts.

Multi-step forecasting

Predict several future periods from one forecast origin. The evaluation setup must match this real deployment requirement.

Common Patterns in Time Series

A series does not have to contain every pattern below. These are common structures to look for before selecting a method.

Trend

A long-run rise, fall or other smooth movement. Trend can be linear or curved.

Seasonality

A repeating pattern tied to a known frequency, such as day-of-week, month-of-year or hour-of-day.

Cycles

Longer rises and falls whose duration is not locked to a fixed calendar period, for example some business or economic cycles.

Irregular / residual variation

Variation not explained by the modeled structure. Residual patterns are a warning that the model may still be missing useful information.

Decomposition: Separate Trend, Seasonality and Residual

A common additive decomposition writes a series as:

Observed = Trend + Seasonal + Residual

Notice that this decomposition does not return a separate “cycle” component. Different decomposition methods make different structural assumptions. In statsmodels, seasonal_decompose is a simple moving-average-based decomposition and should be treated as an exploratory tool rather than a universal truth about the data.

decomposition.py

import numpy as np
import pandas as pd
from statsmodels.tsa.seasonal import seasonal_decompose

# 4 years of monthly data with a known trend + seasonal pattern
dates = pd.date_range("2021-01-01", periods=48, freq="MS")
t = np.arange(48)
seasonal_pattern = np.array([
    -12, -8, -4, 0, 5, 10,
     14,  9,  4, -2, -7, -9
])

sales = 100 + 0.8 * t + np.tile(seasonal_pattern, 4)
ts = pd.Series(sales, index=dates, name="sales")

result = seasonal_decompose(
    ts,
    model="additive",
    period=12
)

print("First 3 observed:", ts.head(3).round(1).to_list())
print("First 3 seasonal:", result.seasonal.head(3).round(1).to_list())
print(
    "First 3 available trend:",
    result.trend.dropna().head(3).round(1).to_list()
)
print(
    "Max absolute residual:",
    round(result.resid.dropna().abs().max(), 6)
)

# Output:
# First 3 observed: [88.0, 92.8, 97.6]
# First 3 seasonal: [-12.0, -8.0, -4.0]
# First 3 available trend: [104.8, 105.6, 106.4]
# Max absolute residual: 0.0

Why is the residual exactly zero here?

Because this teaching series was deliberately created as an exact additive trend plus a repeating 12-month seasonal pattern. Real data is usually noisier, so residuals will not normally be zero.

The Core Evaluation Rule: Preserve Time Order

For ordinary forecasting, the evaluation set should occur after the training data. Randomly mixing future observations into training can produce an unrealistic estimate of performance.

TRAIN: Jan 2021 → Dec 2023

TEST: Jan → Dec 2024

“Never shuffle” is a good beginner rule for ordinary forecasting, but the deeper principle is more precise: your validation scheme must reproduce what information would really have been available at prediction time. In unusual experimental settings, another design may be justified—but it must not leak future information.

Baseline Forecasts — Start Simple

A baseline gives you a minimum standard that a more complicated model should justify beating. The correct baseline depends on the series.

Naive

Forecast future values using the most recent observed value.

Seasonal naive

Forecast using the corresponding value from the previous seasonal cycle.

Mean / moving baseline

Use a historical mean or recent-window summary when that matches the problem.

baseline_forecasts.py

import numpy as np
import pandas as pd
from sklearn.metrics import mean_absolute_error, mean_squared_error

# Same 4-year monthly teaching series
dates = pd.date_range("2021-01-01", periods=48, freq="MS")
t = np.arange(48)
seasonal_pattern = np.array([
    -12, -8, -4, 0, 5, 10,
     14,  9,  4, -2, -7, -9
])

sales = 100 + 0.8 * t + np.tile(seasonal_pattern, 4)
ts = pd.Series(sales, index=dates, name="sales")

# Hold out the final 12 months
train = ts.iloc[:-12]
test = ts.iloc[-12:]

# Baseline 1: repeat the last observed value
naive = pd.Series(train.iloc[-1], index=test.index)

# Baseline 2: repeat the previous 12-month seasonal cycle
seasonal_naive = pd.Series(
    train.iloc[-12:].to_numpy(),
    index=test.index
)

for name, preds in [
    ("Naive", naive),
    ("Seasonal naive", seasonal_naive)
]:
    mae = mean_absolute_error(test, preds)
    rmse = mean_squared_error(test, preds) ** 0.5
    print(f"{name:<15} MAE={mae:.2f} RMSE={rmse:.2f}")

print("Train:", train.index[0].date(), "to", train.index[-1].date())
print("Test: ", test.index[0].date(), "to", test.index[-1].date())

# Output:
# Naive           MAE=14.57 RMSE=16.74
# Seasonal naive  MAE=9.60 RMSE=9.60
# Train: 2021-01-01 to 2023-12-01
# Test:  2024-01-01 to 2024-12-01

Do not memorize “seasonal naive is best.”

It is better only for this particular example. A non-seasonal random-walk-like series may favor a naive forecast, while another dataset may need a different baseline.

Forecast Error Metrics

Forecasting usually uses regression-style error measures because the target is numeric. No single metric is best for every problem.

Metric	Main idea	Useful when	Caution
MAE	Average absolute error	You want an error in the target's original units	Does not increase the penalty quadratically for large misses
RMSE	Square errors, average, then square-root	Larger errors should receive more weight	Can be strongly influenced by large errors
MAPE	Absolute error relative to actual magnitude	Relative percentage-style interpretation is meaningful	Problematic when actual values are zero or very close to zero
MASE	Scale MAE by a naive in-sample error	Comparing error across series with different scales	Requires a sensible scaling baseline and seasonal definition

Worked Example

Suppose:

Actual: 120, 135, 128, 142, 130

Forecast: 118, 140, 125, 145, 133

Absolute errors: 2, 5, 3, 3, 3

Therefore:

MAE

(2+5+3+3+3)/5 = 3.20

RMSE

≈ 3.35

MAPE

≈ 2.43%

forecast_metrics.py

import numpy as np

actual = np.array([120, 135, 128, 142, 130], dtype=float)
predicted = np.array([118, 140, 125, 145, 133], dtype=float)

errors = actual - predicted
mae = np.mean(np.abs(errors))
rmse = np.sqrt(np.mean(errors ** 2))
mape = np.mean(np.abs(errors / actual)) * 100

print(f"MAE:  {mae:.2f}")
print(f"RMSE: {rmse:.2f}")
print(f"MAPE: {mape:.2f}%")

# Output:
# MAE:  3.20
# RMSE: 3.35
# MAPE: 2.43%

MAPE warning

When actual values are zero or very close to zero, percentage errors can become undefined or extremely large. Do not choose MAPE only because percentages sound easy to explain.

Walk-Forward / Time-Series Cross-Validation

One final holdout tells you how the model performed over one future window. During model development, you may want several historical forecast origins. A time-aware splitter repeatedly trains on earlier observations and validates on later observations.

Fold 1: [TRAIN 0–5] [VALIDATE 6–7]

Fold 2: [TRAIN 0–7] [VALIDATE 8–9]

Fold 3: [TRAIN 0–9] [VALIDATE 10–11]

Standard K-Fold is not wrong because it always shuffles—it does not shuffle by default. The problem is that ordinary folds can train on observations that occur after the validation observations. That is why a time-aware splitting strategy is needed for forecasting.

time_series_split.py

import numpy as np
from sklearn.model_selection import TimeSeriesSplit

X = np.arange(12)
cv = TimeSeriesSplit(n_splits=3, test_size=2)

for i, (train_idx, valid_idx) in enumerate(cv.split(X), start=1):
    print(
        f"Fold {i}: train={train_idx.tolist()} "
        f"validation={valid_idx.tolist()}"
    )

# Output:
# Fold 1: train=[0, 1, 2, 3, 4, 5] validation=[6, 7]
# Fold 2: train=[0, 1, 2, 3, 4, 5, 6, 7] validation=[8, 9]
# Fold 3: train=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] validation=[10, 11]

Match the folds to the business horizon.

If production requires a 12-month forecast from a single origin, evaluating only repeated one-month-ahead predictions can give a misleading impression. Validation should imitate the horizon, update frequency and data availability of the real system.

Machine Learning Forecasting with Lag Features

A tabular ML model such as Gradient Boosting does not automatically understand time. We can convert past observations into features such as lag_1, lag_12, rolling statistics and calendar variables.

Target month	lag_1	lag_2	lag_12	month	Target sales
Jan 2025	Dec 2024	Nov 2024	Jan 2024	1	?

ml_one_step_forecasting.py

import numpy as np
import pandas as pd
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.metrics import mean_absolute_error

rng = np.random.default_rng(42)
dates = pd.date_range("2019-01-01", periods=84, freq="MS")
t = np.arange(84)

sales = (
    100
    + 0.7 * t
    + 12 * np.sin(2 * np.pi * t / 12)
    + rng.normal(0, 2, 84)
)

df = pd.DataFrame({"sales": sales}, index=dates)
df["lag_1"] = df["sales"].shift(1)
df["lag_2"] = df["sales"].shift(2)
df["lag_12"] = df["sales"].shift(12)
df["month"] = df.index.month
df = df.dropna()

features = ["lag_1", "lag_2", "lag_12", "month"]

# Last 12 target months are evaluated chronologically
train = df.iloc[:-12]
test = df.iloc[-12:]

model = GradientBoostingRegressor(
    n_estimators=100,
    max_depth=2,
    learning_rate=0.05,
    random_state=42
)
model.fit(train[features], train["sales"])

pred = model.predict(test[features])
naive_pred = test["lag_1"].to_numpy()

print("Train rows:", len(train), "| Test rows:", len(test))
print(
    "Naive one-step MAE:",
    round(mean_absolute_error(test["sales"], naive_pred), 2)
)
print(
    "Gradient Boosting one-step MAE:",
    round(mean_absolute_error(test["sales"], pred), 2)
)
print("First 3 model forecasts:", np.round(pred[:3], 2).tolist())

# Output:
# Train rows: 60 | Test rows: 12
# Naive one-step MAE: 4.58
# Gradient Boosting one-step MAE: 4.38
# First 3 model forecasts: [144.02, 154.19, 155.89]

What exactly did this ML example evaluate?

It is a rolling one-step-ahead setup. For each target month, lag features use actual observations from earlier months. This is appropriate when the real system receives each new actual value before making the next forecast. It is not the same as producing all 12 future months at once from a single forecast origin.

Going Deeper: Multi-Step Forecasting Strategies

Recursive

Predict one step, feed that prediction back as an input, then predict the next step. Errors can accumulate.

Direct

Train separate logic/models for different horizons, such as one-month-ahead and six-month-ahead.

Multi-output

Predict several future horizons together when the modeling method supports it.

There is no universally best strategy. The right choice depends on horizon length, model type, data size and how errors propagate across steps.

Point Forecasts and Forecast Intervals

A point forecast gives one number, such as “next month's demand = 1,250 units.” Real forecasts are uncertain, so many forecasting systems also report an interval.

Point forecast: 1,250 units

Illustrative interval: 1,150 to 1,360 units

An interval is not a guarantee that the future value must lie inside that range. Its interpretation depends on the method and assumptions used to construct it.

Forecasting Methods Compared

Family	Typical idea	Useful when
Naive / seasonal naive	Repeat recent or seasonal history	Baseline and surprisingly competitive simple patterns
Moving averages	Summarize a recent window	Smoothing or simple short-horizon rules
Exponential smoothing	Update level, trend and possibly seasonality	Structured level/trend/seasonal patterns
ARIMA / seasonal extensions	Autoregression, differencing and lagged innovations	Linear time dependence with suitable transformations
Tabular ML	Lag, rolling, calendar and external features	Nonlinear relations or many useful predictors
Sequence / deep learning	Learn complex temporal representations	Large or complex sequence problems where extra complexity is justified

A more complex family is not automatically more accurate. Model choice should be demonstrated by time-aware validation against meaningful baselines.

Common Forecasting Mistakes

Leaking future information

Better approach: Create features and preprocessing so that every forecast uses only information that existed at that forecast origin.

Using a random split by habit

Better approach: Use chronological holdouts or time-aware folds when future periods are the real target.

Skipping a baseline

Better approach: Measure naive or seasonal-naive performance before claiming that a complex model adds value.

Evaluating the wrong horizon

Better approach: If production needs a 12-step forecast, validate a 12-step forecasting process—not only one-step predictions.

Using MAPE near zero

Better approach: Choose another metric when zero or near-zero actual values make percentage errors unstable.

Calling every pattern “seasonality”

Better approach: Seasonality repeats at a known frequency; longer irregular cycles are a different idea.

Treating decomposition as ground truth

Better approach: Decomposition depends on the selected method, period and assumptions.

Assuming ARIMA requires a stationary raw series

Better approach: Differencing is part of ARIMA; the modeled AR/MA structure is applied after the chosen integration/differencing treatment.

A Practical Forecasting Checklist

1. Define the target, data frequency and forecast horizon.
2. Plot the series and inspect trend, seasonality, missing periods and unusual events.
3. Decide exactly what information is available at each forecast origin.
4. Hold out later periods for realistic evaluation.
5. Build at least one simple baseline.
6. Choose metrics that reflect the cost and scale of mistakes.
7. Train and tune models using time-aware validation.
8. Compare models with the same horizon and same evaluation windows.
9. Inspect residuals and forecast plots—not only a single score.
10. Refit/update the forecasting process when new data arrives, according to the deployment plan.

Quick Recap

Why is a random train/test split risky for forecasting?

Because training can contain observations that occur after validation/test observations, giving the model information that would not have existed at prediction time.

Why build a naive baseline?

It tells you whether the extra complexity of your forecasting model produces a meaningful improvement over a simple rule.

Is one-step MAE enough if the business needs a 12-month forecast?

No. The evaluation should reproduce the 12-step forecast horizon, because error behavior can change as the forecast moves farther from the origin.

Connect This Lesson

ARIMA Moving Average Exponential Smoothing Next: Semi-Supervised Learning

Summary

Good forecasting begins with the evaluation design, not with the most advanced algorithm. Define the forecast origin and horizon, preserve time order, understand the series, establish meaningful baselines, and compare models on future-like windows using suitable error metrics.

Key Takeaway

The best forecasting model is not the one with the most sophisticated name. It is the one that consistently improves on sensible baselines under an evaluation setup that matches how the future will actually arrive.

Forecasting Basics