don't let your backtest lie to you: leakage guards for a solo research codebase

lights on the wrong thing.

the backtest printed a return rate north of 1.0 — more money notionally out than in, across a full season the model had never been allowed to see. for about four seconds i believed it. then i did the only responsible thing you can do with a number that flattering when you’re the only person checking your work: i assumed i’d cheated somewhere, and went looking for where.

quincia is a personal statistical-research project with a narrow, slightly masochistic question at its centre: can careful modelling — dixon-coles goal models, a gradient-boosted ensemble, calibrated probabilities — out-guess la quiniela, the spanish football pools, over a season? the pool is pari-mutuel. you’re not betting into fixed odds, you’re splitting a shared prize with everyone else who picked the same fourteen results, so “edge” means being differently right from the crowd, not merely right.

let me put the important part up front and leave it there: this is simulated research. no real money has ever passed through it. the whole system runs paper-only, behind an activation gate that has not been cleared — that gate is guard four below. the match data comes from public football-data sources, pulled read-only and rate-limited. i’m not going to teach you to build a betting system, because i haven’t built one that works. the interesting part was never the model. it’s the scaffolding that stops me from lying to myself.

because a solo research codebase has a structural problem: you are the prosecution, the defence, and the jury at once. every degree of freedom you leave open — a hyperparameter you can nudge, a split boundary you can slide, a data source that quietly overlaps your test set — is a lever that, over enough thirty-second reruns, bends the result toward the answer you were hoping for. nobody does this on purpose. that’s exactly why it has to be structurally impossible, not something you promise yourself you won’t do.

the anti-pattern, in two heads

the failure i was most afraid of has two.

the first is tuning on the test season. you hold out 2025-26, build everything on the earlier years, and then — because the held-out number is right there and rerunning costs nothing — you start making “small” choices while watching it. a draw floor here, an ensemble weight there. each is individually defensible. collectively they’re hand-run gradient descent on the test set, and the season stops being held out the instant its score influences a decision.

the second is independence-approximating your way to optimism. run a parameter sweep, pick the best cell, and quote its p-value as if you’d only ever run one test. or treat fourteen correlated matches on the same matchday as fourteen independent bets when you build a confidence interval. both shrink your error bars below what the evidence supports.

the four guards below exist so neither head grows back. none of them makes the strategy profitable. they make the measurement honest, which is a smaller and far more achievable thing.

guard one: a model that remembers what it was trained on

a trained artefact is just a file on disk. nothing about catboost_model.cbm tells you which seasons it saw. so the moment you add a --skip-train fast path — “don’t retrain, just load the model i fit last time and evaluate it” — you’ve built a loaded gun. load a model that was trained on data now overlapping your test season and it will happily produce a gorgeous, leaking backtest, in total silence.

the fix is to make every artefact carry its own provenance. when quincia trains a model it writes a sidecar <artefact>.meta.json next to it, stamped with a content-addressed hash of the training-relevant config. “content-addressed” is doing real work: the hash digests only the fields that change what ends up inside the model, and nothing else.

src/quincia/provenance.py

# Fields that materially affect what gets written into a trained artefact.
# Adding a new training-relevant config field?  Add it here so the hash
# changes accordingly.
HASHED_CONFIG_FIELDS: tuple[str, ...] = (
    "training_cutoff_season",
    "validation_season",
    "validation_matchday_cutoff",
    "test_season",
    "sel_frac",
    "cal_frac",
    "dc_xi",
    # ... hyper-parameters, calibration flags, retraining schedule ...
)

the hash itself is boring on purpose — canonical json, one sha-256 ↗ , done:

src/quincia/provenance.py

def compute_config_hash(cfg: QuinciaConfig) -> str:
    """Deterministic SHA-256 of the training-relevant config fields."""
    payload: dict[str, Any] = {}
    for field in HASHED_CONFIG_FIELDS:
        value = getattr(cfg, field, None)
        ...
    canonical = json.dumps(payload, sort_keys=True, default=str)
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

the omissions matter as much as the inclusions. paths, artefact filenames, optimiser and staking knobs — all of that lives in an explicit NON_TRAINING_CONFIG_FIELDS set instead. move the project, rename a file, change the staking budget: the hash doesn’t budge, because none of that changes the model. and a test fails loudly if a new config field is ever added without a human deciding which bucket it belongs in — the two sets have to partition every field between them.

then, before any --skip-train reuse, the harness checks the sidecar against the config it’s about to run:

src/quincia/provenance.py

provenance = load_sidecar(artefact_path)
if provenance.test_season != cfg.test_season:
    msg = (
        f"test_season mismatch for {artefact_path}: artefact was "
        f"trained for test_season={provenance.test_season!r}, but "
        f"current config asks for test_season={cfg.test_season!r}.  "
        "Re-train (drop --skip-train) before evaluating."
    )
    raise ProvenanceMismatchError(msg)

a missing sidecar is also a mismatch — absence of evidence gets treated as evidence of a problem. the predict path runs the same guard over every artefact on disk before it loads a single row of data, so a stale cross-season model raises loudly instead of quietly leaking. it’s the same move as the singular-matrix guard from the ols post : make the code refuse rather than lie.

guard two: the calendar is the split

the model must never see the future. in a football season “the future” is precise: any match played after the one you’re predicting. two things enforce it.

first, the split is temporal and keyed to the test season, so held-out is held-out by construction. shift the test season and the training and validation windows slide back with it:

src/quincia/config.py

| Test season | Training cutoff | Validation |
|-------------|-----------------|------------|
| 2025-26     | 2023-24         | 2024-25    |
| 2024-25     | 2022-23         | 2023-24    |
| 2023-24     | 2021-22         | 2022-23    |

the validation season then gets split again, chronologically, into three disjoint blocks — a selection block for ensemble weights, a calibration reservoir for the probability calibrator, and a truly held-out evaluation tail. earlier blocks fit, later blocks judge; never the reverse.

second, and easier to get subtly wrong: features. holding out whole seasons is worthless if, inside the training set, an early match can see stats computed from a later one. so features are built with a per-match cutoff — each target match uses its own date as the wall:

src/quincia/pipeline.py

# Build features with per-match temporal cutoffs to prevent leakage:
# each training match uses its own date, so a 2018 match only sees
# ratings/form/league data from before that date.

mechanically that’s a timeline of rating snapshots and a bisect ↗ to grab the most recent state strictly before the match:

src/quincia/features/builder.py

idx = bisect.bisect_right(sorted_dates, match_date) - 1
if idx >= 0:
    snapshot_date = sorted_dates[idx]
    cached_pi, cached_elo = ratings_timeline[snapshot_date]

bisect_right(...) - 1 is the whole no-lookahead rule in one line: the largest snapshot dated on-or-before the match, and never one after. this is exactly what scikit-learn’s common-pitfalls docs ↗ warn about under data leakage — that any statistic touching held-out rows biases you optimistically — dragged from the preprocessing step into the time dimension. and there’s an end-to-end test whose only job is to spy on every fit call during a 2025-26 backtest and assert none of them ever received a 2025-26 row, with a canary that injects a bogus one to prove the detector is armed.

guard three: correcting for the sweep

say the leakage guards hold and a parameter sweep still turns up a variant that beats the baseline. before believing it, i have to answer: is that the parameter, or is it the best of forty coin-flips?

that’s the family-wise error rate ↗ problem, and quincia handles it with the romano-wolf step-down procedure — from romano & wolf’s 2005 econometrica paper ↗ , titled, aptly, stepwise multiple testing as formalized data snooping. the twist that makes it fit is the resampling unit. a naive bootstrap resamples individual bets and pretends they’re independent; they aren’t, because a whole matchday’s fourteen results share weather, a fixture calendar, an international break. so the bootstrap resamples matchdays, not bets:

src/quincia/backtest/romano_wolf.py

for b in range(n_resamples):
    # Common indices for paired comparisons
    idx_base = rng.integers(0, n_base, size=n_base)
    ...
    for k in range(n_hypotheses):
        ...
        # Recentered delta under H0
        delta_b = (v_rr_b - base_rr_b) - observed_deltas[k]
        boot_abs_t[b, k] = abs(delta_b)

two details carry the honesty. the resample draws matchday indices with replacement and reuses the same draw for the baseline and every variant, so a lucky matchday helps or hurts all of them together — you’re testing the parameter, not the luck. and the delta is recentred by subtracting the observed effect, simulating the null hypothesis that the variant does nothing. the step-down then walks hypotheses from largest effect down, rejecting until one fails to clear its bootstrapped critical value, and enforces monotone adjusted p-values so a weaker result can never look more significant than a stronger one. more power than a flat bonferroni correction, same family-wise guarantee.

guard four: two numbers, not one

here’s where the gambling framing stops being a disclaimer and becomes code. the temptation with a research system is a single threshold: “if the backtest clears X, ship it.” that collapses two very different questions — is this allowed to touch real money at all? and has this achieved what i set out to do? — into one, and the gap between them is exactly where wishful thinking lives.

so there are two numbers, and they are not the same number:

src/quincia/config.py

REAL_MONEY_GATE: float = 1.15
REAL_MONEY_TARGET: float = 1.50

the gate, 1.15, is the activation threshold: the minimum lower-confidence-bound edge required before any real-money play is even considered. it sits deliberately above break-even (1.0), because the interval it’s compared against is conditional — it captures matchday-realisation noise only, not parameter or model-choice uncertainty — so the buffer covers what the interval structurally can’t. the target, 1.50, is the project’s stated ambition. the target is strictly above the gate, and clearing the gate is necessary-but-not-sufficient for calling the project a success.

and the gate defaults to closed:

src/quincia/backtest/gate.py

Semantics:
  - No report on disk → NOT cleared (absence of evidence is not
    evidence of absence; under real-money posture, absence = deny).
  - Report present but `rr_lo_90` missing → NOT cleared.
  - Report present and `rr_lo_90 < gate` → NOT cleared.
  - Report present and `rr_lo_90 >= gate` → CLEARED.

until a report on disk proves otherwise, every euro figure the tool prints comes out wearing a PAPER: prefix under a banner that says, in a box, do not submit these. that gate has not been cleared. everything this project has ever produced is paper. that’s not modesty — it’s the honest state of the numbers.

the honest part: what these guards don’t do

the blog rule around here is that i don’t get to pretend. so:

none of this makes the strategy profitable. leakage guards are not edge — they’re the opposite. they remove the fake edge that leakage manufactures. a perfectly-guarded backtest can, and in the honest case probably does, show that the thing doesn’t beat the pool. the guards’ entire job is to let that bad news through cleanly instead of dressing it up.

they catch my mistakes, not the market’s. the provenance hash catches me reusing a stale model. the temporal split catches me leaking a future match. the romano-wolf correction catches me over-reading a sweep. every one of them is a mirror pointed at the researcher. not one knows anything about whether la quiniela’s crowd is beatable — that’s a fact about the world, and no amount of internal hygiene reveals it.

the correction is only as good as its clustering assumption. the whole matchday-level bootstrap rests on the claim that a matchday is the right unit of correlation — bets within one dependent, bets across them not. if there’s structure i’m not modelling — a team’s form autocorrelating across weeks, say — my clusters are too small and the family-wise guarantee quietly weakens. the correction assumes i got the dependence structure right, and it can’t check that assumption for me. same shape of problem as the collinearity trap : the math is correct, but only for the question you actually asked it.

none of these are bugs. they’re the boundary of what a guard can do. i’d rather state the boundary than let a green test suite imply there isn’t one — the same argument i made in “i said i quit testing, then i shipped seven things” : test the thing that would actually ruin you.

lessons learned

• make leakage structurally impossible, not a promise. “i won’t tune on the test set” is a new year’s resolution. a provenance hash that refuses to load a cross-season model, and a split that slides with the test season, are facts. the difference is whether the guard survives you at 1am with a rerun thirty seconds away.
• the resampling unit is a modelling decision, not a detail. the entire family-wise correction lives or dies on “bootstrap matchdays, not bets.” get the cluster wrong and every downstream p-value is confidently miscalibrated, and nothing in the code looks broken.
• two numbers beat one. splitting “allowed to run” from “actually succeeded” is what stops “looks profitable” from silently becoming “ship it.” the gate is boring, closed by default, and it’s the most important line in the repo.

next up: the pari-mutuel prize model itself — why being right is worthless if the whole crowd was right with you, and the contrarian-value math quincia uses to hunt for results the pool underprices rather than results that are merely likely. it won’t make the strategy profitable either. but it’ll be honest about it.

the project stays paper-only until the gate says otherwise, which is to say: probably forever. that’s fine. the point was never to beat the pool. it was to build something that would tell me the truth when i couldn’t.