Ask a large language model or strategy generation software to design a trading strategy and it obliges instantly. Ask for fifty and it gives you fifty. They’ll look reasonable. Some will backtest beautifully. And that’s exactly where a lot of traders are about to lose money in a brand-new way.
The disconnect is simple once you see it. AI is genuinely good at one half of systematic trading and completely blind to the other half.
What AI does well: generate ideas
This is the language model and strategy generation software’s home turf:
- It has read enormous amounts of trading literature and combines concepts fluently.
- It never runs out of energy. Want twenty variations on a volatility-filtered mean-reversion setup? Done, faster than you could type them.
- For a systematic trader, it collapses the tedious part of research, hand-coding variation after variation, into seconds.
That’s real value, and it isn’t going away.
What AI does badly: know which ideas are real
A language model has no built-in concept of out-of-sample validation. No walk-forward framework. No sense that testing 500 strategies and keeping the best means the best one is probably lucky.
It generates candidates. It does not vet them. And it hands you a gorgeous equity curve with identical confidence whether that curve reflects a durable edge or pure noise fit to the past.
Worse, this danger is bigger than it used to be:
- The core trap in strategy development is the multiple comparisons problem: the more candidates you test on the same data, the more likely some look excellent by chance alone.
- Test enough random rules and the best one always impresses you, edge or not.
- AI and strategy generation software didn’t create this. It poured gasoline on it, because the whole point of these tools is testing far more candidates than you ever could before.
More candidates, more false positives, more beautiful backtests that die live, more blown accounts.
The fix: a two-stage pipeline
Don’t abandon AI generation. Stop treating a good backtest as a finish line and start treating it as an application to be reviewed.
Stage one: generate broadly.
- Let AI do what it’s good at. Cast a wide net across different logic families, timeframes, and conditions.
- Don’t fall in love with anything. A strategy here is a hypothesis, nothing more.
Stage two: validate ruthlessly.Vs Random Testing Create the best possible strategy by random signals, synthetic data. Let the best Vs Random strategy be your benchmark any real strategy must beat. If you cannot beat this benchmark, you may have found something by pure luck and no real edge.
Monte Carlo permutation testing scrambles the data to kill any real signal while keeping its shape, then runs your full selection process on the permutated data many times. If your real result doesn’t clearly beat that noise, the strategy is indistinguishable from luck.
The pairing works because it plays each tool to its strength:
- AI supplies breadth.
- Statistical validation supplies skepticism.
- AI without validation is a false-positive factory. Validation without generation is slow and narrow. Together they’re the best of both worlds.
The mindset shift
The traders who will win with AI aren’t the ones generating the most strategies. They’re the ones who treat generation as cheap and validation as sacred.
Internalize the inversion: in the old world, the hard part was coming up with ideas. In the AI world, ideas are free, compute is cheap, and the hard part is disbelieving them.
Modern strategy-development platforms increasingly build this two-stage discipline in directly, automating both the broad AI-driven generation and the ruthless validation in one workflow, so the skepticism isn’t something you have to remember to apply.
However you assemble it, the principle is the same: let AI generate, then make every candidate survive a gauntlet before it ever sees real capital.
The idea is the easy part now. Proving it is the whole game. Everything can be automated now but you have to know what to automate!
David Bergstrom is founder of Build Alpha, a research platform for systematic traders focused on automatic strategy generation, strategy validation, portfolio construction, and market context. He can be reached at david@buildalpha.com or Buildalpha.com
Twitter: @DBurgh @buildalpha
No position in any of the mentioned securities at the time of publication. Any opinions expressed herein are solely those of the author, and do not in any way represent the views or opinions of any other person or entity.
