Hold back a slice of data the strategy never touched during the build, run the finished rules on it unchanged, and compare the two results as distributions rather than totals. An edge that repeats on data it never saw is tested. Quantprove scores that comparison as a Stability Score, where 60 or above means the holdout behaved like the build.
Data with zero influence on the strategy's design. Not the data you built on, and not the data you glanced at while tuning either, because the moment a result feeds back into a parameter choice, that data joined the build. Out of sample survives only while it stays unseen.
This is the working answer to the question every backtest leaves open: did the rules find a pattern in the market, or a pattern in that particular slice of history? A real edge shows up in both samples. A fitted one stays home.
You only get to read the holdout once.
Chronologically, with the holdout at the end. Keep the most recent 20 to 30% of trades untouched, build everything on the earlier portion, then run the finished rules forward across the holdout one time. A forward split respects how you will actually trade, since live trading is always the newest data, and it stops information from the future leaking backward into the build.
In simpler words... build on the past, grade on the part the build never met. A live paper period or a small size live period works as a holdout too, and it is the strictest one available, because the market generates it after every decision is locked.
A rolling version of the same idea: fit the parameters on one window of data, test them on the next window, slide both forward, repeat. Instead of one holdout verdict you get a sequence of small ones, which exposes parameter fragility, the strategy that only works when fitted to the exact stretch it is about to trade. Quantprove runs walk forward analysis inside Backtest, splitting your record into train and test segments and reporting how the edge held across them.
As distributions, never as totals. The holdout is smaller, so its total R will be smaller, and comparing totals tells you nothing. The questions that matter: did EV per trade hold, did the downside profile keep its shape, did the return distribution stay recognizable. Quantprove's Validation runs this comparison directly, with EV retention, Sortino retention, and an Anderson Darling distribution test, and reports the result as a Stability Score where 60 or above reads as stable.
Sample size honesty applies on both sides: Validation's confidence multiplier keys on the live or holdout sample with no floor, so a thin holdout is discounted hard rather than trusted politely. A holdout gap is also the first diagnostic when live results drift from the backtest later.
Re-running until it passes is the big one. Test, fail, adjust, test again: by the third loop the holdout is just another training set, and its pass means nothing. Picking the best variant by holdout results does the same damage in one move, since the selection itself is a fit. So does retouching costs, filters, or session rules after seeing the test, however small the touch feels.
One honest caveat: almost nobody runs this perfectly, including the people writing about it. The workable standard is not purity, it is bookkeeping. Count your peeks, and let the count discount the result. One look, full weight. Three looks and a tweak, you are reading an inflated number and should treat it that way.
A holdout you peeked at is a backtest with extra steps.
One more test, aimed at a different failure. The holdout asks whether the edge repeats on fresh data. The permutation test asks something harsher about the original result itself: shuffle the trades thousands of times and check whether random ordering produces results as good as yours, which is the direct measure of whether even the in sample edge beats luck. That one deserves its own piece, and it is next in this series.