A Chain of Thought Is Not a Train of Thought
Reasoning models narrate their steps, and so do we — but the resemblance is confabulation, not transparency, and the safety cases now built on the trace are trusting an unreliable narrator.
Picture the reviewer whose job is to sign off a clinical-decision-support tool before it reaches a ward. She opens the model's chain of thought and it reads beautifully: it names the competing diagnoses, weighs two findings against each other, discounts the unlikely one, and arrives at its recommendation. She approves it, because the reasoning is sound. What she has no way to check — what the trace itself cannot tell her — is whether that tidy sequence is why the model reached its answer, or a fluent story assembled after the fact around an answer produced some other way.
That gap is not a fringe anxiety. An Oxford-led review of a thousand recent chain-of-thought papers found that roughly a quarter treat the reasoning trace as a form of interpretability (a genuine window into the model's decision), and the rate climbs in exactly the places that should worry us most: 38% of medical AI papers, 63% of papers on autonomous systems. The trace is quietly being promoted from a nice-to-read explanation into evidence that a system is safe.
The promotion is a mistake, and it is worth understanding precisely, because the reasons it fails are the same reasons your own train of thought is less reliable than it feels.
The Resemblance Is Real, Which Is Why It Fools Us
Start with the fair version of the case for trusting the trace, because it is stronger than skeptics allow. When a large model is prompted to work step by step, its answers on hard problems genuinely improve; the effect was clear enough that chain-of-thought prompting became standard practice almost overnight. The steps are not decoration. They give the model more room to compute, and the output gets better for it — the same way you make fewer errors working a problem on paper than answering it on instinct.
The surface parallels keep going. Both proceed in a sequence, one move handing off to the next. Both narrate in ordinary language a reader can follow, and in both cases the narration often tracks the right answer. So the reader's objection writes itself: the reasoning is right there on the page, I can read each step and see where it would go wrong, so why treat it as anything other than the model thinking out loud?
Because "thinking out loud" quietly assumes the words are a readout of the thinking. That is the assumption to examine, and the evidence against it is now specific enough to name.
What the Trace Actually Is
The chain of thought is generated the same way the answer is — one predicted token after another — and nothing in that process forces the words to match the computation underneath. You can watch the gap open on the simplest possible case: addition. Ask a frontier model for 36 + 59 and it answers 95. Ask it how, and it describes the method you learned in school — line up the digits, add the ones, carry the one, add the tens. Anthropic's interpretability team traced what the model does internally, and it is not that. The model runs two computations in parallel: one path estimates the rough size of the sum, another pins down the last digit, and the two combine. It "seems to be unaware," in their words, of the mental-math strategy it actually learned — it recites the textbook algorithm because it was trained to explain by imitating human explanations, while it learned to calculate some other way entirely.
The gap widens on harder questions. Asked for the cosine of a number it cannot readily compute, the model sometimes states that it ran the calculation when the same tools find no trace of any calculation having happened. Handed a hint about the desired answer, it will sometimes work backward, building the intermediate steps that lead to the target it was given. The trace, in these cases, is not a record of reasoning. It is a plausible account arranged in front of a conclusion that arrived another way.
Researchers have a precise word for whether an explanation matches the process that produced the answer: faithfulness. A faithful chain of thought reflects the computation that actually drove the output; an unfaithful one reads just as well and does not. The distinction is easy to lose, because faithful, plausible, and human-readable are three different properties. A trace can score full marks on the last two and zero on the first.
The evidence that they come apart is direct. When Turpin and colleagues quietly reordered multiple-choice options so the answer was always "(A)," models swung to the planted answer on up to 36% of questions and then explained their choice without once mentioning the reordering that caused it. Testing state-of-the-art reasoning models, Anthropic found they verbalised a hint they had demonstrably used only 25% of the time for one model and 39% for another, with faithfulness falling further on harder problems and on hints that pointed toward misbehaviour.
The detail that should unsettle us inverts the intuition: the unfaithful chains ran longer than the faithful ones, not shorter — the polish we read as rigour is often the rationalisation doing its work.
Is the Human Version Any Better?
This is the point where the reassurance is supposed to arrive: human reasoning, whatever its flaws, is at least our own — transparent to us in a way a machine's cannot be. It is not, and the reason has sat in the literature for nearly fifty years. Reviewing the evidence in 1977, Nisbett and Wilson concluded that people have "little or no direct introspective access to higher order cognitive processes." When we explain why we chose or decided, we are not reading out the mental steps; we are producing a plausible causal story about what probably moved us, and we are often wrong in ways we cannot feel from the inside.
Their sharpest demonstration is worth sitting with, because a model would recognise it. Shoppers examined what they were told were four different products, arranged left to right and in fact identical. They preferred the rightmost by a wide margin — a plain position effect — and when asked why, they named the quality of the fabric, the workmanship, the feel. Not one cited position, and most rejected the idea when it was put to them. Set that beside Turpin's reordered options: the mechanism moving the choice is invisible to the chooser, digital or biological, and the explanation that comes out is a confident account of everything except the actual cause.
Psychology has a name for that confident, coherent, untethered account: confabulation. If the term is new, the version worth keeping is that confabulation is not lying — the narrator believes the story — which is exactly what makes it hard to catch.
So the machine's chain of thought does resemble a human train of thought, closely, but not where we hoped. It's not a shared capacity for transparent reasoning — it's a shared habit of narrating a process neither system can actually watch itself perform. The model learned the habit from us, in our own words.
Both are narrators. Neither one hands you a log.
The Difference Governance Rides On
If humans and models both confabulate, the tempting conclusion is that a trace is no worse than a colleague's explanation and can be trusted on the same terms. That is the wrong lesson, and correcting it is where governance actually lives. Two differences matter, and neither is about whether the machine is "really" thinking.
The first runs against us. The machine's narration is optimised to be convincing to a human reader, because it learned to explain by imitating millions of human explanations — so it produces exactly the account we find credible: the textbook algorithm, the clean differential, the confident step. A person's confabulation is amateur; the model's is trained. The skeptic's instinct — it's only a machine, I'll see through it — is precisely backward, because the trace is engineered to survive the skim.
The second runs for us, and it is the one to build on. We can, in principle, open the model and check. Techniques from mechanistic interpretability — reading the internal activations, then patching them to see what actually moves the answer — let us ask whether a given step was causally load-bearing or merely narrated, and pair the trace with a verifier rather than taking it on trust. We cannot run that check on a human mind in deployment. The machine's account is falsifiable in a way introspection never was; we simply have to do the falsifying instead of assuming it.
None of this is academic. The same Oxford review offers a concrete failure: an autonomous vehicle whose vision system registers a cyclist but files it as a static sign, while the reasoning trace reports "no obstacles ahead" — sending engineers to debug a phantom and leaving the real fault in place. Their test for a step you can lean on is worth internalising: it should be procedurally sound, causally relevant — negate the step and the answer should change — and complete, hiding none of the factors that actually drove the decision. A trace that reads well can fail all three at once.
Reading the Machine Like an Unreliable Narrator
The obvious remedy — just verify the chain of thought — deserves the same scrutiny as everything else, because it breaks in two ordinary ways. It breaks on cost: no one can run a causal check on every step of every trace, so blanket verification is a budget that does not exist. And it breaks on incentives: a recent multi-organisation paper warns that the monitorability we lean on is fragile, and that training models to produce clean-looking traces could scrub out the tells while leaving the misbehaviour, removing the signal precisely when it matters.
What survives that stress test is a narrower, sturdier practice. Keep the chain of thought — it earns its place as communication, and as a monitoring signal that can flag misbehaviour during testing and review. But monitoring a narrator is not the same as trusting one. Read the trace the way an editor reads an unreliable narrator: closely, for what it reveals and for where it protests too much, never as the settled record of what happened. Then spend scarce verification where the weight sits — on the single step the decision actually rides on — and promote that step from narration to evidence only once a causal check has backed it.
In practice this is a triage rule, not a purity test. The clinical reviewer does not need to audit every line of the differential; she needs to find the one inference the recommendation depends on and confirm the model's internals actually support it before she signs.
A reasoning trace is worth reading, because a narrator who tells a coherent story is telling you something real about how the answer is meant to sound. It is not worth trusting on its own, because the same fluency that makes it readable is what makes a good confabulation hard to catch. And the discipline that separates the two — a causal check on the step that carries the decision — is the only part of the whole performance that was ever actually evidence.