I tracked 90 days of LLM sentiment signals against a coin flip

@macro_mikejoined May 8, 2025Aug 2, 2026484 views3 replies

Ninety days ago I built the simplest possible test: an LLM scores the daily headline slate for EURUSD as risk-on / risk-off / neutral, I take only the days where it agrees with my momentum filter, and I compare against (a) the filter alone and (b) a random number generator with the same trade frequency.

Results after 90 days, 41 trades. Filter alone: PF 1.22. Filter + LLM agreement: PF 1.31 on 28 trades. Filter + coin flip: PF 1.19 on 27 trades.

So the LLM "edge" over a coin flip is 0.12 PF on a 28-trade sample — statistically indistinguishable from noise. Where it did help: it kept me out of 3 genuinely ugly central-bank days that the bare filter traded into. Honest conclusion: as a signal it is a coin flip with extra steps; as a veto on event risk it is mildly useful and costs about $2/month.

3 Replies

@chai_quantjoined Sep 18, 2025#1 · 3 days ago
0
Same pattern here: bad signal, decent filter. My LLM use is classifying CPI and central-bank days into a block-list. Measured over 14 months of demo: blocking 8% of trading days cut max drawdown from 21% to 14%. As an entry signal it has never beaten my coin flip either.
One warning from my log: prompt drift. The same prompt scores headlines differently after model version updates, so your "signal" changes without you touching anything. Version-pin the model or log the version with every score.
@grid_goblinjoined Aug 15, 2025#2 · 3 days ago
0
Ran the Twitter/X-sentiment version of this experiment in 2024 and got the same shape: signal equals noise, veto equals useful. The uncomfortable implication is that most "AI edge" products are selling the veto as if it were the signal.
@macro_mikejoined May 8, 2025#3 · 3 days ago
0
Prompt drift is a real one I had not logged — my January scores and March scores may not even be the same instrument. Adding model version to the spreadsheet tonight and re-running the comparison.