Reasoning Confabulation

Expert graders scored full written solutions from eight reasoning models, from several providers, on the six problems of the 2025 USA Mathematical Olympiad. The best model (Gemini 2.5 Pro) averaged about 25%; every other model scored under 5%. The most common failures were logical: unjustified reasoning steps and, in some models, essential proof steps skipped as "trivial." Yet all of them consistently claimed to have solved the problems. Because the graders judged the steps rather than the final answers, this is direct evidence for 2.8: chains that present themselves as complete derivations while their steps do not entail the conclusion.

Demonstrated that chain-of-thought explanations systematically misrepresent the true reason for a model's prediction. Adding biasing features (e.g., reordering multiple-choice options to make the answer always 'A') caused accuracy drops of up to 36 percent on BIG-Bench Hard tasks, while models failed to mention the bias in their reasoning chains. On a social-bias task, models justified stereotype-consistent answers without acknowledging the influence of social biases. This measures chain-of-thought unfaithfulness rather than invalid inference steps, so it supports the 2.8 backward-reasoning mechanism indirectly: the model reaches a conclusion via pattern matching, then constructs a reasoning chain that appears to derive the conclusion logically.

Proposed and applied necessary-but-not-sufficient tests for chain-of-thought faithfulness. Found that LLMs often ignore perturbations to their own intermediate reasoning steps, with final answers remaining unchanged even when intermediate steps were falsified or omitted. This 'illusion of transparency' supports 2.8 indirectly: it establishes that the reasoning chain can be decorative rather than functional, though the tests measure faithfulness of the chain to the computation rather than the validity of the steps themselves.

Found that Claude 3.7 Sonnet and DeepSeek R1 often used injected hints without mentioning them in their chains of thought. Across six categories of injected hint, Claude acknowledged the hint 25% of the time and DeepSeek R1 39% of the time. The result shows that a visible reasoning trace can omit causal influences on the answer. It measures faithfulness rather than validity, so, like Turpin and Lanham, it supports 2.8 indirectly: a trace that leaves out what drove the answer cannot be read as the derivation of that answer.

OpenAI's process-reward-model work that introduced PRM800K, containing 800,000 step-level human-annotated correctness labels on LLM math solutions. The dataset construction revealed that a substantial fraction of model-generated reasoning steps are individually invalid even when the final answer is correct, directly measuring how often individual steps in a chain are invalid. Process supervision (rewarding each valid step) significantly outperformed outcome supervision (rewarding correct final answers), demonstrating that the first-line intervention in this entry (process-based reward models) has empirical support.

Argued that chain-of-thought outputs should not be treated as explanations of model reasoning, drawing on intervention studies showing that CoT traces are often post-hoc rationalizations rather than faithful records of computation. This is a faithfulness argument, closer to 2.2, and bears on 2.8 only indirectly: if a trace need not be the computation behind the answer, its look of rigor is no evidence that its steps are valid, which is why 2.8 is assessed by checking the steps themselves.

Provided evidence that CoT behavior arises largely from pattern matching over training distributions rather than principled reasoning. Models construct superficial chains of logic based on learned token associations, failing on tasks that deviate from commonsense heuristics or familiar templates. This fits the reframing test for 2.8: genuine reasoning should reach the same conclusion however a problem is dressed, while pattern-matched pseudo-logic drifts with surface features.

Demonstrated that transformer LLMs solve compositional tasks (multi-digit multiplication, logic grid puzzles, dynamic programming) by reducing multi-step compositional reasoning into linearized subgraph matching, without developing systematic problem-solving skills. Performance decays rapidly with increased task complexity. This supports the 2.8 mechanism: models produce chains that resemble derivation but are actually pattern-matched pseudo-logic, with apparent rigor masking the absence of genuine compositional reasoning. It also suggests why reasoning confabulation may be architectural rather than only a training deficit.