Reproducibility and experiment discipline

The failure mode this lesson prevents is not dramatic. It is Tuesday’s “0.87 AUC” that cannot be reproduced on Thursday; the plot in the deck whose generating config nobody can name; the improvement that turns out to have been a seed. Reproducibility is treated as virtue; it is actually infrastructure for your own memory — and, with the Evaluation lessons’ error bars, one of the two pillars under every claim you will ever make. The whole discipline reduces to controlling four things: code, data, config, randomness — and writing down which four produced each number.

Randomness: what seeds do and do not buy

Seeding makes pseudo-randomness repeatable. Doing it properly in an ML stack means all of the libraries that keep their own generators:

import random, numpy as np, torch

def seed_all(seed: int):
    random.seed(seed); np.random.seed(seed)
    torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)

# modern numpy style — a passed-around Generator beats global state:
rng = np.random.default_rng(42)

Now the honesty. Seeds do not buy bit-level determinism on GPUs: floating-point addition is non-associative ($(a+b)+c \ne a+(b+c)$ in floats), and CUDA kernels sum in whatever order the hardware parallelism delivers — some (atomics-based scatter ops, certain cuDNN algorithms) are nondeterministic by design for speed. Full determinism is opt-in and costs performance: torch.use_deterministic_algorithms(True) + cudnn.benchmark = False (+ a cuBLAS env var), and some ops simply error. The mature policy has two tiers: bit-determinism for debugging (chasing one divergence) — pay the tax; statistical reproducibility for science — run seeds 0..k, report mean ± the Probability module’s error bars. If a result’s sign flips across seeds, the seed was the result; better to learn that from your own table than from a failed replication. Two more traps worth their sentences: DataLoader workers each need seeding (worker_init_fn, and generator= for the shuffle), and any set/dict-iteration-dependent data ordering is a nondeterminism you planted yourself.

Config: the run’s genome, in one place

Every hyperparameter, path and flag lives in one declared structure — a dataclass or YAML — never scattered across argparse defaults, module constants and notebook cells. The test of adequacy is brutal and simple: could you re-run March’s experiment from its saved config alone? That demands: the config is saved with the run’s outputs automatically (not by hand, because hands forget), it includes the “boring” fields (data version, preprocessing flags, seed — the ones that turn out to matter), and defaults changes are versioned like code, because a silently changed default invalidates every comparison across the change. Tools (Hydra, pydantic) help with composition and validation; the principle — one genome per run, stored where the run’s results live — needs no tool at all.

Tracking: the lab notebook that writes itself

Filenames — model_final_v2_REAL.pt — are where results go to die. An experiment tracker (MLflow and W&B are the defaults; a disciplined runs/<id>/ convention with JSON is a legitimate minimum) records per run: the config, the git commit (plus dirty-diff — uncommitted changes are part of the code state), metrics over time (final numbers hide the learning curves that diagnose), environment lock (last lesson’s artifact), and produced files. What this buys is not compliance — it is queryability: “all runs on dataset v3 where lr < 1e-3, sorted by val AUC” answered in seconds, which changes what questions you bother asking. Three habits make a tracker earn its keep: log immediately (a run not logged at launch never gets logged), name runs by hypothesis (“wd-sweep-0.1” not “run-47”), and record the conclusion in the run’s notes while it is fresh — the tracker stores numbers; the note stores why you ran it.

Data: the fourth axis, and the one that moves by itself

Code is versioned by git; data drifts silently — a re-pulled table, an edited CSV, an upstream schema change, and every historical comparison is quietly void. Minimum viable data discipline: immutable raw data (read-only originals; all cleaning is code from raw → processed, so processed data is derived state, reproducible and deletable), a version identifier in every config (a path convention data/v3/, a snapshot date, or content hashes via DVC/lakeFS when files are large), and split by hash of a stable ID, not by train_test_split(random_state=...) on row order — hash splits survive re-sorting, appends and pipeline reruns, and they close the door on the subtlest leakage: yesterday’s training examples migrating into tomorrow’s test set when the data grows. (The Evaluation and Backtesting lessons each met this door from their own side.)

The lesson — and the module — in one paragraph: pin the four axes (locked environment, committed code, saved config, versioned data + seed policy), let a tracker write the notebook, and every number you produce becomes an object someone can stand on — including, six months from now, you. That standard is what separates an experiment from an anecdote, and it is the default posture of every strong ML team you will join or build.

Exercises