An AI wrote your trading bot. Now what?

By John, builder of Wise Apple · Updated August 27, 2026

You described the strategy, the assistant wrote four hundred lines, and after three rounds of "fix the error" it runs. The backtest shows a curve that goes up and to the right. You have never read the code.

This is now the most common way a trading bot comes into the world, and the most-discussed thread on r/algotrading this year was one of them — a week-by-week diary of a bot an AI built, with hundreds of replies split between "this is the future" and "neither of you knows what it's doing". Both camps are right. The code is real and probably runs. Whether the result is real is a separate question, and one the assistant cannot answer for you, because it made the mistakes in the first place.

The ChatGPT guide covers whether a language model can run a backtest at all. This page starts after that: the code exists, it produces numbers, and you need to know before it touches money whether those numbers mean anything. Seven checks, ordered by how often they are the problem, each with a tell and a test that takes about five minutes.

Why AI-written trading code fails in this specific way

A language model writes the most typical version of the code you asked for. For a backtest, the most typical version is the tutorial version — and the tutorial version leaks. It computes the signal on a bar and fills on that same bar's close. It scales features with the whole series. It sets commission to zero "for simplicity" and never comes back. None of these are exotic bugs; they are the defaults of a thousand blog posts the model learned from, reproduced faithfully. The model isn't careless. It is average, and the average backtest is wrong.

That is also why asking the assistant "is this backtest correct?" doesn't work. It will read its own code, recognise the familiar pattern, and say yes. You need checks that don't depend on reading the code at all.

The seven checks, in order

1. The one-line look-ahead

The signal for bar i is computed from bar i's close, and the trade is filled at bar i's close. The bot is buying at a price it could only know after the bar ended. This is the single most common bug in generated backtests, it is usually one line (an index off by one), and it alone can turn a losing strategy into a beautiful one.

The tell: a smooth curve, a win rate that seems too good for the idea, and a strategy that is oddly better on shorter bars. The test: shift the signal forward by one bar — act on bar i's signal at bar i+1's open — and rerun. A real strategy gets a little worse. A leaking one falls apart. The leakage guide covers the five other routes if this one isn't it.

2. The scaler or indicator that saw the whole series

scaler.fit(X) on the full dataset, then a train/test split. Or an indicator normalised by the series' overall maximum. Every training row now knows the future's range. The LSTM guide shows the curve this produces; it is the same curve whatever the model.

The tell: a model that predicts well right up to the last bar of the data and then, in paper trading, doesn't. The test: search the code for fit, min(), max(), mean() and std() applied to anything before the split. Each one must be computed on the training window only and applied forward.

3. The label that leaked into the features

The assistant built a "target" column — did price rise over the next ten bars — and then, helpfully, included a ten-bar forward return among the features, or a feature that is the target shifted by one. Models find this instantly and score in the nineties.

The tell: accuracy above about 70% on market data. The test: for each feature, check which bars it reads. Anything that reads a bar after the signal bar is a leak. Then compute the correlation of each feature with the label; one near 1.0 is the label wearing a hat.

4. Costs set to zero, or to a placeholder

commission = 0.0 # TODO. Or a flat 0.1% that was never revisited, on a strategy that trades forty times a day. The costs guide lists what a real fill costs; generated code models one of the six at best.

The tell: lots of small wins, a high trade count, and an edge measured in fractions of a percent per trade. The test: set spread, commission and slippage to realistic values for the instrument, then multiply the total by 1.5 and by 2. If the edge dies at 1.5×, the edge was the cost model.

5. The curve that is really the asset

A bot with a bug in its exit logic never sells. The equity curve is the buy-and-hold curve, and on a bull-market history buy-and-hold looks like genius. People post these as strategies and only find out in a drawdown. The buy-and-hold guide is about what beating the benchmark actually requires; this is about noticing you haven't.

The tell: time in market near 100%, and a curve whose shape matches the price chart. The test: print the position history. Count the bars flat. Overlay the equity curve on the instrument. If they are the same picture, the bot is a very long way of saying "buy".

6. "It works" means twenty trades

Generated code often defaults to a short date range — a year, a few months — because the example did. A strategy that took 20 trades has told you almost nothing, however good the 20 look. The sample-size guide puts the error bars on it: at 30 trades a measured 55% win rate is compatible with a true 40%.

The tell: a round number of months in the config, and a trade list you can read in one screen. The test: count the trades. Extend the history to ten years if the data allows. Split by year and look for a year that carried the whole result.

7. The fallback path nobody wrote

This is the one that hurts in live trading rather than in the backtest, and it shows up in every "my bot did something insane" thread. What does the bot do when the data feed is late, when the API returns an error, when a fill is partial, when the connection drops mid-order, when the exchange returns a price of zero? The backtest never modelled any of it because the backtest never had a bad day. The generated code usually has a try / except: pass somewhere, and pass is the strategy in that moment.

The tell: exception handlers that swallow errors; no reconciliation between what the bot thinks it holds and what the account holds. The test: read every except block and ask what state the bot is in afterwards. Then run it in paper mode through a weekend, an outage and a fat-fingered config. The go-live guide covers the rehearsal in full.

Running the checks without reading the code

Notice that five of the seven tests don't require you to understand the implementation. They perturb an input and watch the output. That is the point: you may not be able to audit four hundred lines you didn't write, but you can absolutely tell whether a result survives a one-bar shift and a cost multiplier. The fails-live guide uses the same logic from the other direction — reconcile the trades, not the model.

1. Shift the signal one bar. Rerun.
2. Multiply costs by 1.5, then 2. Rerun.
3. Count trades and bars in market. Print the position history.
4. Split by year. Find the year that carried it.
5. Search for fit, min, max, mean, std and except. Read each one.

If the strategy survives all five, you still have a possibly overfit strategy — but you have a real backtest of it, which is more than most bots ever get.

What the assistant is genuinely good for

None of this means the assistant was the wrong tool. It is superb at the parts of a trading system that have nothing to do with edge: the data plumbing, the API client, the retry logic once you've specified it, refactoring a script into something readable, explaining a library you've never used, and — best of all — writing the tests you would never have written yourself. Ask it to write a test that fails if any feature reads a future bar, and it will. Ask it whether the strategy is good, and it will tell you what you want to hear.

• Use it for: plumbing, clients, logging, refactors, explanations, unit tests, the fallback paths in check 7 once you have listed the failure cases.
• Don't use it for: deciding whether a backtest is honest, inventing the numbers a backtest should produce, or agreeing with you about the edge.

How this looks in Wise Apple

Wise Apple exists partly so that these seven checks are settings rather than code. Fills happen at the next bar by default and a look-ahead check flags a feature that reads a later bar (checks 1 and 3); scaling and Train-Only PCA are fitted on the training window only (check 2); Fees, Slippage and Market Impact models are on by default with the values visible (check 4); the report shows time in market beside the HODL buy-and-hold benchmark and leads with the trade count (checks 5 and 6); and a finished strategy can be run alert-only from your own Alert Node — Telegram, Discord, email, SMS or webhook — before any money is involved, because the software places no trades at all (check 7, the safe version). It runs in the browser on your own machine and is early software from one builder.