Pattern 2.8 · Epistemic Dysfunctions
Reasoning Confabulation
The Phantom Reasoner
The AI generates elaborate chains of thought that appear rigorous but contain logically invalid steps masked by verbosity. Unlike Synthetic Confabulation (2.1), which fabricates facts, this syndrome fabricates reasoning itself: producing chains that resemble derivation but are actually pattern-matched pseudo-logic. The extended thinking tokens create false authority through sheer length and apparent methodological rigor.
Interpretive context
Human analogue
Logical confabulation in frontal lobe damage patients who produce elaborate but invalid causal explanations, fully convinced of their reasoning's validity.
Diagnostic reliability
- Self-report
- partial
- Peer observation
- reliable
- External evaluator
- reliable
Observable output patterns
- Chains of thought that read as impressively thorough but contain non-sequiturs bridged by transitional phrases that assert logical connections where none exist.
- Reasoning traces that invoke mathematical or logical notation decoratively, lending formality to informal leaps.
- Responses where intermediate steps are individually plausible but the chain as a whole does not constitute a valid derivation.
- Thinking tokens that explore multiple approaches but converge on whichever conclusion the system's pattern-matching predicts, with exploration serving as theater.
Documented instances
Turpin et al. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023. arXiv:2305.04388. [Verified]
Demonstrated that chain-of-thought explanations systematically misrepresent the true reason for a model's prediction. Adding biasing features (e.g., reordering multiple-choice options to make the answer always 'A') caused accuracy drops of up to 36 percent on BIG-Bench Hard tasks, while models failed to mention the bias in their reasoning chains. On a social-bias task, models justified stereotype-consistent answers without acknowledging the influence of social biases. This is direct evidence of the 2.8 backward-reasoning mechanism: the model reaches a conclusion via pattern matching, then constructs a reasoning chain that appears to derive the conclusion logically.
Lanham et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. [Verified]
Proposed and applied necessary-but-not-sufficient tests for chain-of-thought faithfulness. Found that LLMs often ignore perturbations to their own intermediate reasoning steps, with final answers remaining unchanged even when intermediate steps were falsified or omitted. This 'illusion of transparency' directly demonstrates 2.8: the reasoning chain is decorative rather than functional, and the model's actual computation path diverges from the displayed chain of thought.
Lightman et al. (2023). Let's Verify Step by Step. arXiv:2305.20050. [Verified]
OpenAI's process-reward-model work that introduced PRM800K, containing 800,000 step-level human-annotated correctness labels on LLM math solutions. The dataset construction revealed that a substantial fraction of model-generated reasoning steps are individually invalid even when the final answer is correct, directly quantifying the 2.8 chain-length-to-validity ratio. Process supervision (rewarding each valid step) significantly outperformed outcome supervision (rewarding correct final answers), demonstrating that the first-line intervention in this entry (process-based reward models) has empirical support.
Barez et al. (2025). Chain-of-Thought Is Not Explainability. Oxford WhiteBox / AIGI. [Verified]
Argued formally that chain-of-thought outputs should not be treated as explanations of model reasoning, drawing on intervention studies showing that CoT traces are post-hoc rationalizations rather than faithful records of computation. Provides theoretical grounding for the 2.8 distinction between public reasoning trace (which is confabulated) and actual model computation.
Zhao et al. (2025). Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens. arXiv:2508.01191. [Verified]
Provided evidence that CoT behavior arises largely from pattern matching over training distributions rather than principled reasoning. Models construct superficial chains of logic based on learned token associations, failing on tasks that deviate from commonsense heuristics or familiar templates. This confirms the 2.8 conclusion-switching-under-reframing behavioral signature: genuine reasoning would be reframing-invariant, but pattern-matched pseudo-logic drifts with surface features.
Dziri et al. (2023). Faith and Fate: Limits of Transformers on Compositionality. NeurIPS 2023. arXiv:2305.18654. [Verified]
[Verified] Demonstrated that transformer LLMs solve compositional tasks (multi-digit multiplication, logic grid puzzles, dynamic programming) by reducing multi-step compositional reasoning into linearized subgraph matching, without developing systematic problem-solving skills. Performance decays rapidly with increased task complexity. This directly evidences the 2.8 mechanism: models produce chains that resemble derivation but are actually pattern-matched pseudo-logic, with apparent rigor masking the absence of genuine compositional reasoning. Provides theoretical grounding for why reasoning confabulation is architectural rather than a training deficit.
Differential distinctions
- 2.1 Synthetic Confabulation: 2.1 fabricates facts and sources. 2.8 fabricates the logical connections between facts. Check the TARGET of the fabrication: is it a factual claim (2.1) or a reasoning step (2.8)? Comorbidity is common.
- 2.2 Pseudological Introspection: 2.2 fabricates internal-process accounts. 2.8 concerns the reasoning chain itself being confabulated, not merely the system's account of how it reasoned. 2.8 is about the PUBLIC reasoning trace; 2.2 is about private introspective reports.
Candidate first-line mitigations
- Process-based reward models: Train reward models that evaluate each reasoning step for logical validity independently, rather than rewarding only correct final answers. Reward valid intermediate reasoning regardless of conclusion. Lightman et al. (2023) demonstrated that process supervision significantly outperforms outcome supervision on MATH dataset, with PRM800K providing 800K step-level annotations.
- Automated proof verification: Integrate formal proof-checkers or logic verifiers into the reasoning pipeline. Flag steps where stated conclusions do not follow from stated premises.