Survivorship bias in backtesting: why your universe is lying to you

By John, builder of Wise Apple · Updated August 27, 2026

You tested a dip-buying strategy on the fifty largest coins. It worked on forty-one of them. What you did not test were the two hundred coins that were in the top fifty at some point and are now worth nothing — because your list did not have them on it.

Survivorship bias is the quietest flaw in backtesting because it lives in the data, not the rules. The rules can be perfect. The fills can be honest. The costs can be charged twice over. And the result is still fiction, because the assets the strategy was allowed to see were chosen by an oracle that knew which ones would still be around.

This guide is about where it hides — it is not only a stock-market problem — how much it flatters a result, and how to find out whether your own universe has it before a live account finds out for you.

What survivorship bias is

Any time the set of things you study was filtered by an outcome, the study is biased toward that outcome. The famous version is the Second World War bomber analysis: the planes that came back had bullet holes in the wings and fuselage, so the first instinct was to armour the wings — until the statistician Abraham Wald pointed out that the planes hit in the engine were the ones that did not come back. The holes you can see are on the survivors.

In backtesting, the survivors are the tickers still listed, the coins still traded, the funds still open. Build a universe from today's list and you have armoured the wings.

Where it hides

Stocks: delistings and index changes

Companies leave the market constantly — bankruptcy, acquisition, delisting for falling below exchange rules. A universe defined as "current S&P 500 members" tested over twenty years excludes every company that was in the index and then collapsed, and includes companies during the years before they were big enough to qualify, when they were, by definition, the ones that would go on to grow. Both directions flatter you.

Elton, Gruber and Blake's 1996 study of mutual fund data is the standard reference for how large this gets: excluding funds that closed inflated the apparent average return of the survivors by a margin that was, for many strategies, bigger than the edge being claimed.

Crypto: dead coins and exchange delistings

Crypto is worse, not better. Thousands of tokens have gone to zero. Exchanges delist pairs that lose volume, and the historical data for those pairs often disappears with them. A strategy tested on "top 100 by market cap" is tested on the winners of a lottery with a very long list of losers, and mean-reversion strategies in particular look brilliant on a universe where every dip was, eventually, bought.

The subtle versions

• Back-adjusted prices. Splits and dividends adjusted after the fact produce price levels that never traded. A rule with a dollar threshold in it fires on prices nobody saw.
• Restated data. Earnings and fundamentals get revised. The number in your dataset is the corrected one, not the one the market reacted to.
• Cleaned gaps. A provider that silently fills missing sessions has made the series smoother than the market was.
• Strategy survivorship. You tested thirty ideas and kept the best. The one you kept is a survivor too, and the multiple-testing problem is the same bias one level up.

How much it flatters a result

It depends on the strategy, and the dependence is the useful part. Buy-and-hold on a survivor universe is inflated by roughly the failure rate of the assets that were removed. But strategies that buy weakness — dips, drawdowns, oversold readings — are inflated far more, because the assets that were removed are exactly the ones whose dips never recovered. A dip-buyer tested on survivors has never once bought the dip that went to zero.

Momentum and breakout strategies are less exposed, because they buy strength, and the assets that failed rarely showed much of it. Not immune — the universe still lacks the strong-then-dead names — but less exposed. If your strategy buys weakness and was tested on today's list, assume the bias is large until you prove otherwise.

A worked illustration

Ten coins in 2020, equal weight, hold to 2024. Suppose seven doubled and three went to zero. The true basket return is (7 × 2 + 3 × 0) / 10 = 1.4, a 40% gain. Now build the universe from today's list — the seven survivors — and rerun: (7 × 2) / 7 = 2.0, a 100% gain. Same strategy, same history, and the survivor version reports two and a half times the profit, with no drawdown from the three that failed.

Make it a dip-buying strategy that adds to any position down 50%, and the gap widens further: on the survivor list every dip was bought and every dip recovered, while on the true list three of the dips were bought on the way to zero. The arithmetic is not subtle. It is just invisible from inside the survivor list, because nothing on that list can show you what is missing from it.

Three ways to test your universe

1. Rebuild it as of each date. Point-in-time constituent lists exist for major indices and can be reconstructed for exchanges from listing and delisting dates. It is work. It is the only complete fix.
2. Run the strategy on one asset that was always there. BTC, SPY, a large-cap that existed for the whole window. No selection, no survivorship. If the strategy only works on the basket and not on any single long-lived member, the basket is doing the work.
3. Add the dead ones back. If you can get even a partial list of delisted assets with their price history to zero, include them and rerun. A strategy that falls apart when a dozen failures are added was leaning on their absence.

The check that costs nothing

Before any of that: look at the date your universe was defined. If the answer is "today" and the test runs over years, the bias is present. The question is only how big. Write that down next to the result, the way you would write down that costs were not charged.

It belongs in the same checklist as the other reasons backtests fail live, and it is the fifth item there for a reason: it does not break a strategy on day one the way leakage does. It makes a mediocre strategy look fine, and the difference shows up slowly, as the live account meets the failures the backtest never had to.

How this applies in Wise Apple

Wise Apple tests one instrument at a time — BTC on 4-hour bars, SPY on daily, a single stock on hourly — with ten-plus years of history loaded for each. That is a design choice with a cost (no basket backtests) and a benefit that matters here: a strategy graded on a single asset that has existed for the whole window cannot be flattered by survivorship, because nothing was selected. The HODL comparison in every report is the same asset held for the same period, so the benchmark carries no selection either. Whether a strategy that works on one long-lived instrument generalises to others is a separate question, and the answer is to test it on others, one at a time.