What Walk-Forward Grading Is

The problem: a model can memorize the answer

A machine-learning model is very good at one thing — finding patterns in the data you hand it. If the data you hand it includes the games you are about to grade it on, it will find those too. The resulting score measures memory, not prediction, and it will be excellent.

This is not a rare failure. It is the default outcome of the obvious way to build a backtest: fit on the whole history, then evaluate on the whole history. Everything downstream of that — the win rate, the return, the equity curve — is describing a model that already knew the results.

Walk-forward: train on the past, grade on the next season

Walk-forward grading fixes the order of operations to match reality. You cut the history into chronological folds — usually seasons — and step through them:

1. Train on seasons up to N.
2. Predict season N+1, which the model has never seen, and record every prediction.
3. Move the boundary forward one season, retrain from scratch, and repeat.

The retrain in step 3 is the part people skip, and it is the whole point. A model trained once on everything and then replayed season-by-season is still a model that saw the future; only refitting at each boundary keeps each fold's prediction honest. What you end up with is a record of out-of-sample predictions, made in the order they could actually have been made.

This is how our own backtests run — one model per season fold, retrained at each boundary, so no fold is ever scored on data it was fit on. It is also slower and produces worse-looking numbers than the alternative, which is the honest cost of the method.

Three ways a backtest still lies to you

1. Leakage: the answer is hiding in a feature

Splitting by season is necessary but not sufficient. Leakage is any path by which information from after the decision point reaches the model — and it is usually subtle, not a duplicated column.

A concrete example from our own build, because it is the clearest illustration we have: our model Studio grades a user's trained model against the closing line, and for a period it was handing the model the closing line itself as an input feature. The model learned to repeat the number it was about to be measured against, and the grade looked strong. It was measuring an echo. We shipped a leakage gate that excludes close-derived features from a run and annotates the result when it fires.

What that cost, measured: on an NBA moneyline test, dropping the close-derived features took the beat-the-close rate from 0.535 (95% CI 0.521–0.549) to 0.496 (95% CI 0.481–0.511) — a coin flip. The entire apparent result was the peek. We publish that number because it is the most useful thing we learned that quarter, and because a leakage story with no before-and-after is just a reassurance.

The general rule: for every feature, ask when was this value knowable? If the answer is "after the moment I would have placed the bet," it cannot be in the model. Closing lines, final injury designations, and anything derived from the result all fail that test.

2. No control: 50% is not the null hypothesis

A backtest that reports 54% correct sounds like a finding. It usually is not, because sports markets are not coin flips and "half" is the wrong thing to compare against. Home teams win more than half their games. Favorites win most of theirs. A model that has quietly learned "pick the home side" will clear 50% comfortably while containing no information whatsoever.

The fix is a control that is settled on the same games, at the same real prices, so it is a like-for-like comparison rather than a rhetorical one. Ours are:

• Always home / always over — take side A in every graded game. Exposes a result that is really just a home or over bias.
• Random side — pick a side per game from a seeded, deterministic coin flip, so the same backtest re-run produces the identical control number. This is what "no information at all" scores at these prices, and it is meaningfully below zero, because of the vig.
• Always favorite (moneyline) — take the shorter price every time. A cheap check that a positive result is not just favorite-riding.

A model result is only interesting to the extent it separates from those. Our backtest report renders the headline next to the controls for exactly that reason — and when a run predates the controls, it says so instead of presenting a number without them.

3. A return figure with no denominator you can inspect

Return on a backtest is the most quoted number and the least stable one. It moves with stake sizing, with which games you chose to count as bets, with the start and end date of the window, and with whether the vig was modeled on the real per-side prices or waved through. Two honest analysts can grade the identical set of predictions and print returns that differ by several points, none of it because the model changed.

That does not make the number useless — it makes it a number you can only read next to its controls and its sample size. Read alone, it is decoration.

Why we report a rate against the close

The headline we hold our models to is the beat-the-close rate: the share of graded predictions where the price at prediction time was better than the market's final price. Three reasons it survives scrutiny better than a return figure:

• It has no stake-sizing knob. A rate is a count over a count. There is no bankroll assumption to tune until the chart looks good.
• It is unit-safe. Spread and total closing line value is measured in points; moneyline CLV is measured in cents. Those do not average together, so a cross-market "average CLV" is a category error. A rate counts the same way in every market.
• It is hard to cherry-pick. Every window we publish carries its win, loss, and push counts alongside the rate. A rate that quietly drops its losers is a highlight reel, and the counts are what stop it from being one.

The thing a beat-the-close rate is not is a profit statement. The sportsbook's margin sits between getting a better number and coming out ahead, and a rate above half does not clear that margin on its own. CLV is evidence about your process; it is not a forecast of your bankroll. Anything that presents it as one has changed the subject.

What a passing grade looks like — and what ours is

We hold a model architecture to a two-part bar on a walk-forward holdout: a beat-the-close rate above 0.52, and a positive average CLV within a single market's native unit. Two parts, because either one alone is gameable — a rate can drift above half on tiny margins, and an average can be carried by a handful of outliers.

The honest state of that scoreboard, stated the same way on our methodology page: no model of ours clears the vig against the close yet. The signal we measure on moneylines is real and it is not large enough to pay for the juice by itself. We say that on every model surface, because a backtest you cannot trust is worse than no backtest, and a research tool that only reports its good folds is not a research tool.

A checklist for reading anyone's backtest

Including ours. Six questions, in the order they matter:

1. Was the model retrained at each fold? If it was fit once on everything, the chart is memory.
2. When was every feature knowable? Anything from after the decision point is leakage, however innocuous the column name.
3. What did a no-information control score on the same games? If that number is missing, the result has no scale.
4. How many bets, and over what window? A great-looking rate on 40 graded games is noise wearing a percentage sign.
5. Are losses and pushes in the denominator? Ask what got excluded and why.
6. Is the vig priced in at the real per-side numbers? A result that only works at prices nobody offered is a simulation of a different market.

Where Blitzen fits

Blitzen is a research terminal for bettors who do their own work — a tool, not a picks service. On grading specifically:

• You can run the walk-forward yourself. Our backtesting engine grades over historical seasons with per-fold retraining, and returns the report with its control baselines and CLV metrics attached rather than a single number.
• The report is built to show a null result. Headline next to always-home, random-side, and always-favorite, with an explicit note when a legacy run has no controls to show. Losses and pushes are in the counts.
• Models you train are graded the same way. Our model Studio — in beta, under published caps — trains a light model on your feature weights and grades it walk-forward against the close, with the leakage gate on and the excluded features listed in the result.

None of that is a system for getting ahead of a sportsbook. It is the apparatus for finding out whether an idea holds up once you stop letting it see the answer. Most do not. That is the useful part.

Read the market. Then check its work.

Blitzen is an analytics and research tool, not betting advice or a picks service. Nothing here predicts a financial outcome. Sports betting carries real financial risk — wager only what you can afford to lose, and only where it is legal. Must be 21+. If gambling stops being fun, help is available: call 1-800-GAMBLER.