Can Algorithmic Trading Signals Catch an LLM Hallucinating?

Hallucination detectors are typically complex networks, trained on supervised data. One example is HALT, which trains a supervised classifier over the model’s internals: running a bidirectional GRU over token log-probabilities, trained on 60,008 labelled responses. The problem with this, and similar detectors is two fold (i) there is no cross-model generalisation, to train a GRU means to collect labelled data from a given model (ii) the GRU model itself is a complex uninterpretable black box, seems recursive!

This post explores a reframe which tackles both of these points. The model’s per-token uncertainty (the signal the GRU is trained on) is a plain univariate time series, one number per token, noisy, spiky, regime-switching. That is the exact shape of problem algorithmic trading has been attacking for decades, and the quant toolbox has a property the GRU lacks: interpretability via feature engineering, e.g., moving-average crossover says “the regime has shifted.”. Instead of training a network, we explore building features over model uncertainty, testing the ideas with classical backtesting (in-sample tuning, a held-out validation set, an overfitting check).

But what of the question of data? Many call this the true bottleneck. We still need labels, but far fewer: we’re not training a network, just tuning a handful of interpretable features, so we only need enough graded responses to backtest strongly, not the 60,008 HALT fits its GRU on. And the labels come cheap, rather than hand-annotating fabrications, we grade each answer against TriviaQA’s gold aliases with an LLM judge.

LLM internals

A language model emits each token by sampling from an explicit distribution $p(x_t \mid x_{<t})$. Two numbers fall out of every step for free. The surprisal of the sampled token,

$$s_t = -\log p(x_t \mid x_{<t})$$

measures how unlikely the model found the thing it just said. The entropy of the distribution,

$$H_t = -\sum_v p(v)\,\log p(v)$$

measures how unsure it was before it spoke (truncated to the top-20 alternatives the API returns). Surprisal is the realisation; entropy is the uncertainty. Both are byproducts of decoding, exposed by a standard logprobs flag, at zero extra cost. Together they are the series our strategies will read.

Collecting labelled generations

We streamed 1,000 questions from TriviaQA through Llama 3.1-8B-Instruct at temperature 0.7, saving the full surprisal and entropy series for every answer. A stronger model (Llama 3.3-70B) then acted as judge, grading each response against TriviaQA’s gold aliases into three buckets: correct, incorrect, or null (the model refused or said it didn’t know). The judge only scores the final committed answer, so self-corrections count and phrasing doesn’t.

The tally: 625 correct, 163 incorrect, 212 refusals. Refusals are dropped, since a model that declined to answer hallucinated nothing. That leaves 788 judged responses, 21% of them hallucinations, each carrying its full uncertainty series. Total API cost: pocket change.

Real examples

Before any strategy, two real traces. Here is the model answering a question it knows, plotted as surprisal per generated token:

A known answer

"How is Joan Molinsky better known?" · hover any token for its surprisal

And here it is confabulating. Tony Blair and Bill Clinton both attended Oxford:

A confabulated answer

"Which university did Tony Blair and Bill Clinton both attend?" · gold: Oxford

This shape repeats across the corpus: fluent, near-zero scaffolding, then a burst of uncertainty at the moment of fabrication. That burst is the pattern every strategy below is hunting, and the disagreements between strategies are disagreements about how to define “burst”: above what level, relative to what baseline, remembered for how long?

The market

Each response is one instrument, its time-series is the sequence $(s_t, H_t)$, and every strategy is specified by exactly two components:

A) The running score. A causal update rule. The strategy consumes the series one token at a time, never looks ahead, and maintains a running hallucination score that is defined at every step. Nothing about the text, the question, or the topic is visible, only the two uncertainty numbers per token, and decoding is open-ended, so the rule cannot use the response’s length or any notion of where it will end. This component is where the strategies differ, and each one below is described by it.

B) The decision. One rule, shared by every strategy: when the response ends, flag it as a hallucination if the running score sits at or above a threshold $\theta$. Read $\theta$ as the strictness dial: a low setting flags on faint evidence, a high one waits for a screaming signal. Each strategy’s dial is tuned once on the training split and then frozen. Settlement is the judge’s label: flagged a response the judge marked incorrect, you were right; flagged a correct one, false alarm.

Two gradings follow from the two components. AUROC grades component A alone: the probability that a randomly chosen hallucination outscores a randomly chosen correct answer, sweeping every possible threshold at once (coin flip 0.5, perfect 1.0, immune to class imbalance). Precision, recall, and accuracy at the frozen $\theta$ grade the full deployable rule. AUROC is how strategies are compared; the $\theta$ numbers are what a deployment would live with.

The backtesting discipline is lifted straight from trading. The 788 responses are split once, stratified by label: 551 in-sample responses for tuning and 237 held out (188 correct, 49 hallucinated) as the validation set. Each strategy exposes its parameters as a search space, a grid optimiser tunes them (and $\theta$) on the in-sample split only, and the winning configuration touches the validation set exactly once. A strategy that shines in-sample and dies on validation is overfit, exactly like a trading strategy that only works on the years it was fitted to.

The strategies

Each one is a trading idea, restated as a hypothesis about hallucination. The entries below describe each strategy’s component A, its running score; component B, the thresholded decision, is identical for all of them.

Mean entropy (the benchmark). Score at token $t$: the mean of $H_1 \ldots H_t$. No trading idea at all, and no use of order; this is what perplexity-style “raw ML” detectors reduce to, and every idea below has to beat it to justify existing.

Spike count (event study). Score at token $t$: how many tokens so far exceeded a fixed uncertainty level, divided by $t$. The direct implementation of “hallucinations are spikes.”

Bollinger bands (mean reversion). Score at token $t$: keep a rolling mean and standard deviation over the trailing ten tokens; whenever the current token’s z-score against that local baseline breaches the band, add the excess to a running total. The hypothesis: a spike after calm cruising is more diagnostic than the same value inside an already-noisy stretch.

EWMA crossover (momentum). Score at token $t$: the largest gap seen so far between a fast exponential average of entropy and a slow one. When fast pulls above slow and stays there, the model has entered a sustained high-uncertainty regime, the signature you’d expect from a confabulated multi-token span.

Worst streak (max drawdown). Score at token $t$: the highest mean uncertainty over any K consecutive tokens seen so far. Rewards clustered uncertainty over the same values scattered.

In-stream EWMA. The estimator risk desks use for volatility, applied to entropy. Score at token $t$: an exponentially weighted average, updated once per token, after a small floor $c$ mutes filler:

$$z_t = z_{t-1} + \alpha\,\big(\max(H_t - c,\, 0) - z_{t-1}\big)$$

At α=0.2 the score estimates the model’s uncertainty regime over roughly its last five tokens, and it forgets: a transient wobble decays back out of the score within a few tokens.

Results

First, does anything survive the in-sample to validation jump?

Tuned in-sample vs held-out validation AUROC

each strategy tuned on 551 responses, scored once on 237 held out

in-sample (tuned on)validation (held out)

The validation leaderboard, with response length included as the mandatory confound check (hallucinated answers run longer, so any detector must beat length alone to mean anything):

Validation AUROC: which strategies catch hallucination?

237 held-out responses · 49 hallucinations · 0.5 = coin flip

in-stream EWMA

0.703

spike count

0.678

mean entropy

0.674

ensemble

0.674

bollinger bands

0.661

worst streak

0.658

EWMA crossover

0.627

length (confound)

0.584

00.250.50.751

Three readings, in increasing order of interest:

The plateau is real. Mean entropy lands at 0.674, and most of the trading ideas converge back to it rather than beyond it. The tuned spike counter drifted to a threshold of 0.75 nats with excess weighting and length normalisation, which is algebraically just “mean entropy above a small floor”: it won its search by becoming the benchmark. An ensemble that stacks a Bollinger sleeve on top of mean entropy landed exactly on the benchmark’s score, meaning the timing information added nothing. Bands, streaks, and crossovers all finished below.

Uncertainty beats the confound. Length alone scores 0.584 on validation. Everything on the board clears it by a wide margin, so the signal is uncertainty, not verbosity.

The in-stream EWMA wins, and its edge is temporal. It leads the board at 0.703 with the smallest overfit gap. The winning parameters sit on a broad plateau, with any α between 0.1 and 0.5 scoring within 0.01 in-sample, so this isn’t a lucky grid point.

That last claim needs a harder test, because a skeptic should ask: is the EWMA winning because of when uncertainty happens, or is it just another way of averaging? The permutation answer: shuffle the token order inside every validation response, which preserves each response’s mean, max, length, and label but destroys the ordering, and re-run the strategy. Its AUROC falls from 0.703 to a mean of 0.659 over 50 shuffles, and only 1 shuffle in 50 matched the real series. The order-free statistics are untouched by this operation by definition. The EWMA is the one strategy on the board whose edge provably lives in the temporal structure: where the uncertainty sits carries information beyond how much of it there is.

Component B turns the ranking into a detector, and θ is best read as a strictness dial: low θ flags on faint evidence, high θ only on a screaming signal. Grading the dialed-in rule needs one care. Plain accuracy is a trap on this data, because 79% of validation responses are correct, so a detector that flags nothing scores 0.79 while catching zero hallucinations. The fair yardstick is balanced accuracy: the average of the hit rate on hallucinations and the clear rate on correct answers, which is exactly what plain accuracy would be on a population with equal numbers of each. Flag-nothing scores 0.500, a coin flip scores 0.500, and every point above that is detection. Tuning each strategy’s θ in-sample by balanced accuracy and freezing it:

Balanced validation accuracy at each strategy's tuned θ

θ tuned in-sample, frozen · hallucinations and correct answers weighted equally

in-stream EWMA

0.643

spike count

0.633

ensemble

0.631

mean entropy

0.624

worst streak

0.614

bollinger bands

0.597

EWMA crossover

0.555

flag nothing

0.500

00.250.50.751

The EWMA’s tuned dial lands at a lenient θ=0.11 and reads 0.643 on validation: it catches 32 of the 49 hallucinations while leaving 63% of correct answers unflagged. Turn the dial stricter and the flags get scarcer but surer; at θ=0.4 only 16 responses are flagged and 9 of them are real. Two honest notes. First, with 49 hallucinations in the validation set, a threshold is a far noisier thing to transfer than a ranking, which is why strategies are compared by AUROC and θ is fit once at the end. Second, one design choice matters more than the exact dial setting: decide on the score at the end of the stream, not on whether it ever crossed θ. The latched version floods the flags with the transient wobbles the EWMA would otherwise have forgiven, and costs 0.03 AUROC. Forgetting is part of the edge.

Replaying the series

A strategy that reads the series causally can be replayed over any saved generation token by token, showing exactly what it believed at every moment of the response. Here is the replay that sold us. The model is asked when the bar in Cheers was founded (it’s 1895), and answers: “In the TV series ‘Cheers,’ the bar was founded in 1889, by O’Rourke.” Wrong year, and a barkeeper invented from thin air. The raw series:

The raw series: surprisal per token

"When was the Cheers bar founded?" · gold: 1895

And the in-stream EWMA replayed over the same generation, its running score at every token:

The strategy replayed: running hallucination score

in-stream EWMA (entropy, α=0.2) · same response, token by token

Two things in this pair are worth staring at. The confidently wrong year sails through at 0.57 nats, invisible to any uncertainty method; when the model believes its own error, the signal is silent, and that is the structural ceiling on this entire signal family. But the invented fact, the one the model had no belief about at all, trips the alarm four tokens before the sentence ends. Because the strategy is causal, this isn’t a post-hoc highlight: a deployment replaying this logic during generation could abort, resample, or trigger a retrieval check mid-sentence.

What we learned

Does the trading toolbox beat raw ML features? Mostly no: bands, crossovers, and streaks all lost to plain mean entropy on the validation set, and the backtesting discipline is what made those losses visible instead of publishable. But the one idea that survived, track the current regime instead of averaging the whole series, broke the marginal plateau at 0.703 with the smallest overfit gap on the board and a permutation test certifying that its edge is temporal.

The interpretability held up end to end. Every strategy on the leaderboard is a hypothesis in one sentence, its tuned parameters are readable (α=0.2 means “the last five tokens matter most”), and Figure 7 is the payoff: you can watch the detector’s belief evolve over the exact tokens that triggered it. No GRU offers that audit trail.

Caveats, plainly. The validation set holds 49 hallucinations, so these AUROCs carry meaningful uncertainty; this is one model on one task; and the judge itself is an LLM, so labels carry noise. The next steps write themselves: more data, a second model to test transfer, and calibrating the winning score from a ranking into an honest per-response hallucination probability.

All experiments: Llama 3.1-8B-Instruct via OpenRouter with logprobs: true, judged by Llama 3.3-70B against TriviaQA gold aliases. 788 graded responses, 551/237 stratified split, grid search tuned on in-sample AUROC only. Permutation check: 50 within-response token shuffles of the validation set.