Backtesting and forecast evaluation

The Supervised ML module closed with evaluation because everything before it was hostage to measurement. Time series doubles the stakes: the i.i.d. assumptions under k-fold cross-validation are exactly false here — the data are ordered and dependent, and shuffling them into folds lets the model peek at the future it claims to predict. Most forecasting systems that fail in production were evaluated with subtly future-leaking pipelines and never knew. This lesson is the immune system.

Rolling-origin evaluation

The only honest simulator of deployment: pick a training cutoff (the origin), fit on data up to it, forecast $h$ steps ahead, score against what actually happened, advance the origin, repeat. Every forecast is made with strictly historical information — by construction, the way production will make it.

Averaging $|e_{o,h}|$ over origins, per horizon, gives the object a naive average hides: the error-vs-horizon curve, which tells you how far ahead your model is actually useful.

Design choices worth being deliberate about: expanding vs sliding window (expanding uses all history — the default; sliding adapts under drift at a variance cost); refit cadence (refit every origin is the gold standard; refit weekly is often the honest match to production); and gaps — if features need $k$ days to arrive in production, the backtest must respect that same delay.

The leakage bestiary, time-series edition

Leakage here is sneakier than in the supervised module, because time gives it more doors:

• Shuffled CV — random folds put future observations in training. The ur-sin. (Sklearn’s TimeSeriesSplit exists precisely for this.)
• Full-series preprocessing — normalizing, imputing, detrending or seasonal-adjusting using statistics of the WHOLE series before splitting: the scaler has seen the future’s mean. All preprocessing goes inside the rolling loop, fitted per origin.
• Feature lag optimism — a “same-day” covariate (weather, prices, traffic) that in production arrives with delay, or revised: backtests use the final revised value, deployment gets the first estimate. Use vintage/as-of data where revisions exist.
• Target-adjacent features — rolling means computed with windows that include the current (to-be-predicted) value; off-by-one in a shift() is the most common single bug in forecasting pipelines.
• Hyperparameter leakage — tuning on the same rolling windows you report. Keep a final untouched holdout period, exactly like the supervised module’s test-set discipline.
• Survivorship — evaluating only on series that lived long enough to have full histories (dead products, delisted stocks silently removed).

Metrics that survive scale

Averaging raw errors across series of different magnitudes lets the big series vote with the loudest voice, and percentage errors (MAPE) explode near zero and asymmetrically punish over-forecasts. The field’s repair:

— error in units of “how much better than the obvious baseline”, scale-free by construction, defined even at zero actuals, comparable and averageable across series (it is the M-competitions’ backbone metric for these reasons). One convention note: for non-seasonal series the denominator is the one-step last-value naive instead — Hyndman & Koehler define both, and the baseline should always be named when a MASE is quoted. Below 1 you are adding value; above 1 the naive method is beating your model. For probabilistic forecasts — which lessons 2–5 all argued you should be producing — score the distribution: pinball loss per quantile, its integral CRPS, and calibration coverage checks (your 90% intervals should contain ~90% of actuals; the Kalman lesson showed how confidently wrong a mis-specified model’s bands can be). A point metric on a quantile system evaluates a shadow of what you built.

Close the loop with the same words the supervised module ended on, now with time attached: the backtest is a simulation of deployment, and every place it differs from deployment — information timing, refit cadence, metric, series population — is a place reality is licensed to disappoint you.

import numpy as np, pandas as pd
from sklearn.model_selection import TimeSeriesSplit

y = pd.Series(np.random.default_rng(0).standard_normal(200)).cumsum() + 50

H, results = 12, []
for origin in range(60, len(y) - H):
    train = y[:origin]                      # strictly past
    scale = train.diff(12).abs().mean()     # MASE denominator, in-sample
    fc = np.repeat(train.iloc[-1], H)       # naive — swap in your model here
    err = (y[origin:origin + H].values - fc)
    results.append(np.abs(err) / scale)

mase_by_h = np.mean(results, axis=0)
print(np.round(mase_by_h, 2))               # the widget's curve, in code

Exercises