In-sample vs. out-of-sample testing that actually holds

By John, builder of Wise Apple · Updated August 27, 2026

You split the data 80/20, trained on the 80, scored on the 20, and got 71%. It felt rigorous. On market data, that split has at least two leaks in it — and the score is partly the leaks.

Out-of-sample testing is the one idea everyone in strategy research agrees on: tune on some data, judge on other data. The disagreement — and the damage — is in the details. Which data. Split how. With what gap. Consulted how many times. Get those wrong and the "out-of-sample" score is an in-sample score wearing a disguise.

This guide separates three things people call the same thing, explains why time-series data breaks the splitting habits learned elsewhere in machine learning, and covers the two tools that keep an out-of-sample result honest: the purge/embargo gap and the permutation test. It ends with the reason the best of many strategies always looks good, and what to do about it.

Three things people call the same thing

The middle column is where most research lives, and it is fine — as long as you remember that every comparison you make against it is a small optimisation. Twenty comparisons later, the out-of-sample set has been optimised on, and only the sealed holdout is still clean.

Why a random split is wrong for market data

In most machine learning, the training and test sets are chosen at random, and that works because the rows are independent — one photograph of a cat tells you nothing about the next. Market bars are the opposite. Two problems follow.

It trains on the future

A random split puts Tuesday in training and Monday in test. The model learns Tuesday's outcome and is then asked about Monday, whose features are almost identical to Tuesday's. It is not predicting Monday. It is remembering Tuesday. Chronological splitting — everything before a date is training, everything after is test — is the minimum, not a refinement.

Adjacent rows are near-duplicates

Even with a chronological split, the rows at the boundary are related. If your label is "did price rise over the next 10 bars?", the label for the last training bar depends on bars that are in the test set, and the first test bars have features (moving averages, lookback windows) built from bars that are in the training set. The split is clean on paper and contaminated in practice.

Purge and embargo gaps

The fix, formalised by Marcos López de Prado in Advances in Financial Machine Learning (2018), is to remove the contaminated rows at the boundary:

• Purge: drop training rows whose labels reach into the test period. If labels look ahead 10 bars, the last 10 training rows go.
• Embargo: additionally drop the first stretch of the test period, so that test rows whose features were built from training bars are excluded too. The embargo is usually sized to the longest lookback in your features.

The cost is a few rows of data. The benefit is that the test score stops including the boundary's free hints — and on short label horizons with long lookbacks, those hints can be most of the score. López de Prado's book also describes combinatorial purged cross-validation, which builds many purged train/test combinations from the same history to get several out-of-sample estimates instead of one; conceptually it is walk-forward with more permutations of which windows count as test.

The permutation test

An out-of-sample score tells you the model did well on unseen data. It does not tell you whether "well" is better than what a model would do on nothing. A permutation test answers that, with no algebra:

1. Take your real training data and shuffle the labels, so each row keeps its features but gets a random outcome. The relationship between features and labels is now destroyed.
2. Run your entire pipeline — optimisation, model training, the lot — on the shuffled data, and score it out of sample.
3. Repeat many times, each with a fresh shuffle. You now have a distribution of out-of-sample scores that your procedure produces on pure noise.
4. Compare your real score to that distribution. If it sits far outside the noise scores, your features contain information. If it sits inside them, your procedure is good at finding patterns in nothing, and it found one. How many shuffles you need depends on how far outside the noise you want to be able to say the real score sits; the sample-size guide has the same square-root logic.

This test is the most direct measurement of the question that matters — is there anything here? — and it is skipped far more often than it is run, because it requires running the pipeline dozens of times. If your pipeline runs on your own machine at no marginal cost, there is no excuse.

Multiple testing: why the best one always looks good

Line up a hundred people and have each flip a coin ten times. Someone will get eight or nine heads. That person has no skill; they were the best of a hundred, and the best of a hundred coin flippers looks skilled by construction.

Now line up a hundred strategy variants and score each out of sample. The best one looks good for the same reason. This is not a flaw in out-of-sample testing. It is what selection does to any test, and the only defences are to count the trials and raise the bar accordingly. Harvey, Liu and Zhu (2016) argued this for the academic literature on return predictors; Bailey and López de Prado's deflated Sharpe ratio (2014) turns the trial count into an explicit discount on the winner's Sharpe. At your desk the practical version is simpler: the number of things you tried is part of the result, and "best of forty" is a claim that needs forty times the evidence.

Putting it together

1. Split chronologically. Never at random.
2. Purge training rows whose labels reach into the test window; embargo the first stretch of the test window.
3. Use the out-of-sample set to compare candidates, and count every comparison.
4. Run the permutation test on the candidate you like. If it is not clearly outside the noise, stop.
5. Walk it forward to check the procedure across several windows — the walk-forward guide covers sizing the windows.
6. Open the sealed holdout once. Report what it says, alongside the trial count.

If the result survives all six, it is worth the next test, which is time: freeze it and let new bars arrive. And if the holdout number is better than the out-of-sample number — better, not merely similar — suspect a leak before you celebrate.

How Wise Apple splits data

Wise Apple's Train/Test Split is chronological by construction, and Embargo Bars inserts the gap between the training and test windows, sized by you to the label horizon and feature lookback. Train-Only PCA and feature scaling are fitted on the training window alone. The Walk-Forward Window rolls the split forward and reports each out-of-sample window separately. There is no built-in permutation test — but because a run costs nothing beyond your own machine's time, shuffling labels via a feature and rerunning is an afternoon rather than a budget line, and the Random Seed setting makes each rerun reproducible.