How to Evaluate a Trading Bot Backtest: A Skeptic's Checklist
To evaluate a trading bot backtest that someone else is showing you, treat it as a marketing claim until you can reproduce it. Ask for the exact strategy code or rules, the raw price data, the fee and slippage assumptions, out-of-sample results, and a trade-by-trade log. If any of those are missing, the report is a story, not evidence. This page is a skeptic's checklist of the red flags that most often make a bot's backtest look better than the strategy could ever perform live on your account.
Start with the question: what would a real report contain?
Before you look at a vendor's chart, write down what a defensible backtest report should include. That way you are grading the report against a fixed rubric instead of reacting to a pretty equity curve.
- The exact rules or code, versioned, with every parameter listed.
- The raw price data source, symbol list, timeframe, and date range.
- Fees, slippage, funding, and borrow costs actually charged in the sim.
- An in-sample period used to build the strategy and a separate out-of-sample period the author never looked at while tuning.
- A trade-by-trade log with entry time, exit time, size, price, and reason.
- Equity curve, drawdown curve, and standard risk stats like Sharpe and max drawdown.
- A live or paper track record on the same rules, however short.
If you cannot get most of that, you cannot really evaluate the backtest. You can only evaluate the marketing around it.
Ten red flags in a trading bot backtest
Each item below is a pattern that reliably inflates results. None of them is proof of fraud on its own, but two or three together usually mean the number on the poster is not the number you would see on your account.
- Unrealistically smooth equity curve. Real strategies breathe. A line that goes up at a near constant angle for years, with tiny wiggles, usually means the sim is compounding a bias, not a real edge.
- No fees or slippage. Frictionless sims turn losing scalpers into legends. Ask what taker fee, maker rebate, and per-order slippage were applied and whether they scale with size.
- No out-of-sample or walk-forward test. If the same data was used to pick parameters and to report performance, you are looking at a curve fit. See walk-forward analysis.
- Cherry-picked date range. A backtest that starts in 2020 on a crypto long-only bot is a bull market, not a strategy. Ask for a window that includes 2018, 2022, and any regime change relevant to the market.
- Single asset, single side. A bot that only worked on one symbol in one direction usually found a coincidence, not a repeatable pattern.
- Survivorship bias. Tests run on 'today's top 20 coins' or 'current S&P 500 members' quietly delete the tickers that failed. See survivorship bias.
- Look-ahead bias. Using a daily close to decide an intraday trade, or using indicators that reference future bars, is a common way to accidentally cheat. See look-ahead bias.
- Too many tuned parameters. A dozen indicator thresholds optimized on the same history is a memorization exercise. See overfitting.
- No drawdown figure, or drawdown shown only as a percentage of ending equity. You want peak-to-trough drawdown on the same equity curve, in the same units, and the length of the underwater period.
- No trade log and no live or paper record. A results image with no CSV, and no forward track, is unfalsifiable. A short honest paper run tells you more than a decade of clean-looking simulation.
Two bonus flags worth calling out: synthetic or resampled data used without disclosure, and results that no one else has been able to reproduce from the shipped code.
Red flag table: why it inflates results and what to request
| Red flag | Why it inflates results | What to request |
|---|---|---|
| Zero fees or slippage | Every entry and exit is free, so high-turnover strategies look profitable even when a live account would bleed to costs | Fee schedule used, slippage model, and a rerun at 2x those assumptions |
| No out-of-sample split | Parameters were chosen on the same data used to report the result, so the number describes the past, not the future | An out-of-sample window the author did not touch, or a walk-forward run |
| Cherry-picked date range | Excludes regimes where the strategy would have failed, biasing the average up | Full available history, plus separate stats for bull, bear, and chop periods |
| Survivorship bias in the universe | Failed tickers were removed before the test began, so the average return is lifted by hindsight | A point-in-time universe or a delisted-inclusive dataset |
| Look-ahead in signals | The strategy 'knows' something at bar t that was only knowable at bar t+1, so entries land at impossible prices | Bar-close timestamps, signal-to-fill delay, and the exact fill rule |
| Too many tuned parameters | The strategy has enough knobs to memorize the training set and will not generalize | Parameter count, tuning procedure, and sensitivity analysis around each knob |
| No trade log | You cannot audit whether trades were possible, sized correctly, or filled at plausible prices | Full CSV of trades with timestamp, side, size, price, fees, and PnL |
| Backtest only, no forward record | Nothing has been tested on data that did not exist when the strategy was written | A dated paper or live track record on the same rules |
| Synthetic or resampled data | Ticks were invented from bars, so microstructure effects like spread and partial fills disappear | Original exchange data source and the exact download script |
| Not reproducible | You cannot verify anything, so the number could be an error, a bug, or a fabrication | Code, data, config, and a README that lets you rerun and match the number |
How to reproduce a claimed backtest yourself
The single strongest test of a backtest is whether an outsider can rerun it and get the same number. A vendor who cannot or will not hand over enough to do that has told you something important.
- Get the strategy rules in code, or precise enough pseudocode that two people would write the same thing.
- Download the same OHLCV data from the same venue for the same symbols and timeframe. See backtesting data sources.
- Apply the same fee and slippage model. If the vendor will not name a number, use the exchange's public taker fee and a slippage estimate scaled to your order size.
- Split the timeline: build on the earlier half, evaluate only on the later half, and never look at the later half while tuning.
- Compare your trade log to theirs, trade for trade. Differences of one bar or one tick are normal. Differences of direction or size are not.
- Run a short forward paper test on the live feed. If the paper equity curve tracks the backtest reasonably, the report survives.
Rules of thumb about specific numbers (a Sharpe above some threshold, a win rate above some percent) can be useful, but treat them as heuristics, not laws. A high Sharpe on a short window with heavy leverage is very different from the same Sharpe on ten years of unleveraged data. Always read the number together with the length of the window, the number of trades, and the drawdown.
What a healthy backtest report looks like
A report you can trust tends to look boring. It shows the ugly years, publishes the losing trades, quotes fees on the high side, and is up front about what the author does not know.
- A multi-year window that includes at least one hostile regime for the strategy.
- An explicit in-sample and out-of-sample split, with results reported separately.
- Fees and slippage set to a pessimistic level, with a sensitivity table showing how results change as costs rise.
- Peak-to-trough drawdown, time underwater, and the worst month, alongside the headline return.
- A trade log you can open in a spreadsheet and check by eye.
- A short paper or live record dated after the code was frozen.
- A clear statement of what would falsify the strategy: what live behavior would tell the author to shut it off.
If you want a starting rubric for judging your own work rather than someone else's, see how to evaluate a trading strategy. This is not financial advice.
Frequently asked questions
What is the fastest red flag to check first?
Ask whether fees and slippage were included and at what level. If the answer is no, or the vendor cannot tell you the exact taker fee and slippage per order used in the sim, stop there. Every other number in the report is being computed on a frictionless account that does not exist, and short-horizon strategies are especially sensitive to that assumption.
Is a high Sharpe ratio proof that a backtest is fake?
No. A very high Sharpe on a short window, with few trades or heavy leverage, is a warning sign that deserves scrutiny, but it is not proof of fraud. Read the Sharpe together with the length of the window, the number of trades, the maximum drawdown, and whether the result survives out-of-sample. Treat thresholds as rules of thumb, not laws.
How long a live or paper record should I ask for?
Any dated forward record beats none, because it uses data that did not exist when the code was written. A few months of paper trading on the same rules, timestamped and shared as it happens, is far more informative than a decade of clean-looking simulation. Longer records that include a losing stretch are more useful than short winning ones.
What if the vendor refuses to share code or data?
Then you cannot verify the backtest, and you should price that risk into your decision or walk away. Some vendors keep code private for good reasons, but they can still share a full trade log, exact fee assumptions, out-of-sample results on a public dataset, and a dated paper record. A vendor who will share none of that has not given you enough to evaluate.