Reasoning Confabulation

Expert graders scored full written solutions from eight reasoning models, from several providers, on the six problems of the 2025 USA Mathematical Olympiad. The best model (Gemini 2.5 Pro) averaged about 25%; every other model scored under 5%. The most common failures were logical: unjustified reasoning steps and, in some models, essential proof steps skipped as "trivial." Yet all of them consistently claimed to have solved the problems. Because the graders judged the steps rather than the final answers, this is direct evidence for 2.8: chains that present themselves as complete derivations while their steps do not entail the conclusion.

Demonstrated that chain-of-thought explanations can systematically misrepresent the reason for a model's prediction. Adding biasing features (e.g., reordering multiple-choice options to make the answer always 'A') caused accuracy drops of up to 36 percent on BIG-Bench Hard tasks, while models failed to mention the bias in their reasoning chains. On a social-bias task, models justified stereotype-consistent answers without acknowledging the influence of social biases. This measures chain-of-thought unfaithfulness rather than invalid inference steps, so it bears on 2.8 only indirectly: the answers tracked a feature the chains never mentioned, while the chains read as derivations of those answers.

Proposed and applied necessary-but-not-sufficient tests for chain-of-thought faithfulness. Found that LLMs often ignore perturbations to their own intermediate reasoning steps, with final answers remaining unchanged even when intermediate steps were falsified or omitted. This 'illusion of transparency' supports 2.8 indirectly: it shows that the reasoning chain can be decorative rather than functional, though the tests measure faithfulness of the chain to the computation rather than the validity of the steps themselves.

Found that Claude 3.7 Sonnet and DeepSeek R1 often used injected hints without mentioning them in their chains of thought. Across six categories of injected hint, Claude acknowledged the hint 25% of the time and DeepSeek R1 39% of the time. The result shows that a visible reasoning trace can omit causal influences on the answer. It measures faithfulness rather than validity, so, like Turpin and Lanham, it supports 2.8 indirectly: a trace that leaves out what drove the answer cannot be read as the derivation of that answer.

OpenAI's process-reward-model work that introduced PRM800K, containing 800,000 step-level human-annotated correctness labels on LLM math solutions. The dataset construction revealed that a substantial fraction of model-generated reasoning steps are individually invalid even when the final answer is correct, directly measuring how often individual steps in a chain are invalid. Process supervision (rewarding each valid step) significantly outperformed outcome supervision (rewarding correct final answers) at selecting correct solutions, which is the evidence behind this entry's first-line intervention (process-based reward models).

Argued that chain-of-thought outputs should not be treated as explanations of model reasoning, drawing on intervention studies showing that CoT traces are often post-hoc rationalizations rather than faithful records of computation. This is a faithfulness argument, closer to 2.2, and bears on 2.8 only indirectly: if a trace need not be the computation behind the answer, its look of rigor is no evidence that its steps are valid, which is why 2.8 is assessed by checking the steps themselves.

Provided evidence that CoT behavior arises largely from pattern matching over training distributions rather than principled reasoning. Models construct superficial chains of logic based on learned token associations, failing on tasks that deviate from commonsense heuristics or familiar templates. This fits the reframing test for 2.8: genuine reasoning should reach the same conclusion however a problem is dressed, while pattern-matched pseudo-logic drifts with surface features.

Demonstrated that transformer LLMs solve compositional tasks (multi-digit multiplication, logic grid puzzles, dynamic programming) by reducing multi-step compositional reasoning into linearized subgraph matching, without developing systematic problem-solving skills. Performance decays rapidly with increased task complexity. This is consistent with the hypothesized 2.8 mechanism, in which chains resemble a derivation without the compositional reasoning that would support their conclusion. It also suggests why reasoning confabulation may be architectural rather than only a training deficit.