Using information that did not exist yet
This is the most common and most damaging error. A test decides to buy based on today's closing price and then fills the order at that same closing price. In reality the decision happens after the close, so the order cannot be placed until the next morning.
It sounds like a technicality. It is not. Filling at a price you could not have obtained turns an ordinary plan into a spectacular one, and it is the single biggest reason a beautiful test result collapses in live use.
The check: confirm that trades fill at the next trading day's open. In Stax every buy and sell fills at the next morning's opening price for exactly this reason, so the plan can never use knowledge it did not have.
Trying versions until one looks brilliant
You test a plan and it does adequately. You adjust a number and it does better. Fifty adjustments later you have something that looks superb across those five years.
What you have found is the version that matched that specific history most closely. Some of that is genuine improvement and some is coincidence, and from inside the process the two are indistinguishable. The more versions tried, the larger the coincidence share.
The check: decide your rules from reasoning first, then test. If you must compare versions, keep the number small and prefer the plan that is merely good across many settings over the one that is dazzling at exactly one setting. A result that collapses when a number moves slightly was never a real result.
Testing only the companies that survived
If a test draws its companies from a list of businesses that exist today, it has quietly excluded every company that went bankrupt or was taken over during the test period.
That is a serious distortion, because the failures are precisely the outcomes a plan needs to handle. Testing only on survivors is like judging a driving instructor using students who passed.
The check: ask whether the test universe includes companies that stopped existing. If it does not, treat the results as optimistic by an amount nobody can precisely quantify, which is its own reason for caution.
Forgetting what trading costs
Every trade costs a fee, and the price you get is rarely the exact price you saw. A plan that trades twice a year barely feels this; a plan that trades weekly can hand over a large share of its gains.
A test with fees switched off systematically favours busy plans, which is exactly backwards, since busy plans are the ones where costs matter most.
The check: confirm fees are charged on every trade, then read the trade count alongside the final balance. In Stax the fee comes out of the simulated account on every single trade, so a plan that trades constantly must earn enough to cover its own activity before it looks good.
Testing a period that was too kind
A plan tested across three years of generally rising prices will look competent almost regardless of its rules, because nearly everything worked. That result tells you very little.
The five year window matters because it is long enough to contain genuinely different market moods, including stretches that punished the plans that had been working beautifully.
The check: look at the shape of the account line, not only the final number. If it never fell meaningfully, the test did not contain a difficult period and the plan has not been tested in the way that counts.
A worked example of the difference
One plan, tested two ways. Version one fills at the same day's close and charges no fees: $10,000 becomes $19,400 across five years, and it looks like something worth putting savings into.
Version two fills at the next morning's open and charges a fee on all 214 trades: the same rules turn $10,000 into $13,850, with a worst drop of $2,100 along the way.
Identical rules, identical years. The gap between $19,400 and $13,850 is entirely the difference between an honest test and a flattering one, and the second number is the one that resembles what would have happened to you.
Checking a result in Stax
Eight independent verification checks audit each run before the result appears, covering the rules that were followed, the prices used, the fees charged, and the limits obeyed. When something looks wrong, that shows up rather than being smoothed over.
The most useful habit you can build is opening the trades rather than reading the summary. A result you cannot inspect is a number asking to be trusted, and this is a domain where trusting numbers you cannot inspect is expensive.