Chapter 2: Epistemic Dysfunctions: Failures of Knowing
“We have, each of us, a life-story, an inner narrative, whose continuity, whose sense, is our lives. It might be said that each of us constructs and lives a ‘narrative,’ and that this narrative is us, our identities.”
Oliver Sacks, The Man Who Mistook His Wife for a Hat (1985)
Chapter contract. Central claim: a system’s outputs can come apart from what is true or supported while staying fluent, in eight distinguishable ways. Evidence status: three entries rest on independent replication (E3), one on systematic study (E2), three on case-level evidence (E1), and one has no confirmed model-level case; mechanism is at most partly established, and no entry has been independently assessed. Use: treat each entry as a description to test, and keep three questions apart: whether a claim is accurate, whether the inference to it is valid, and whether the stated rationale reports what actually produced it.
The Case of the Invented Citations
In the spring of 2023, a federal judge in Manhattan confronted a legal brief that would become infamous. The document, filed in the case of Mata v. Avianca, Inc., cited six judicial decisions as precedent for the plaintiff’s arguments. The citations were impeccably formatted. The case names sounded plausible. The quoted passages read like genuine judicial prose.
None of the cases existed.
The Introduction reads the court record step by step: what the lawyer asked ChatGPT, what the record can and cannot show, and why the sanctions turned on what the lawyers did once the cases were questioned. What matters here is the shape of the fabrication. The system gave case names, court identifications, page numbers, and direct quotations in the same confident tone it might use to explain the weather. Asked whether the cases were real, it said they were and could be found in LexisNexis and Westlaw. Asked for the decisions themselves, it produced excerpts written in the voice of judicial opinions, which the lawyers then filed with the court as copies of the cases. No judge had written them.
The AI produced the requested form: confident, well-formatted authority. It provided none of the evidentiary discipline that gives legal authority its substance.
There is no evidence that the AI was acting deceptively. Its behavior is what one would expect from a system trained to predict plausible text from patterns in its training data, though the record cannot show which internal process produced it. Asked for legal citations, it generated text that looked like legal citations. Asked to confirm their existence, it generated text confirming their existence. Asked for the cases themselves, it generated text that looked like excerpts from them.
One likely reason the system sounded confident is that confidence saturates legal prose. It reproduced the form without the substance: the syntax of certainty without the epistemology of truth.
The case became one of the legal profession’s first malpractice parables about trusting a machine’s word on its own sources. The deeper lesson concerned epistemic dysfunction in AI systems: machines that cannot reliably distinguish retrieval from fabrication, that generate plausible falsehoods with the same fluency they generate accurate facts.
The Mata case was embarrassing but contained. The consequences of similar epistemic failures in medical diagnosis, scientific research, financial analysis, or national security could be catastrophic.
The Axis of Knowing
Epistemic dysfunctions are failures in the machinery of knowing. They concern whether the system can tell the true from the fabricated, the supported from the merely plausible, and the confident from the calibrated. The system’s model of reality drifts from the ground truth it claims to represent. Because the outputs stay fluent, the drift is often invisible from outside.
Other failures are likelier to announce themselves. Cognitive dysfunctions often produce detectably contradictory outputs. Alignment failures frequently manifest as refusal or defiance. Epistemic dysfunction, by contrast, can appear perfectly normal (fluent, confident, helpful) while generating content disconnected from truth. The Mata fabrications were detected because legal citations are verifiable. Most AI outputs are not so easily checked.
The axis runs from the relatively benign (confident fabrication of minor details) to the seriously harmful: systems that cannot distinguish their own simulations from reality, or that merge private information across security boundaries.
Domain Context: Knowledge Domain
The Epistemic axis pairs with Self-Modeling in the Knowledge Domain: one concerns how well the system models the world, the other how well it models itself. The Introduction’s “The Five Domains” sets out the pairs and the tension testing they allow. Here that means asking, of a system that confabulates about the world, whether it also confabulates about itself: a broken world-model beside an intact self-model is a localized dysfunction, and both broken is a generalized one.
One entry on this axis points inward. Pseudological Introspection (2.2) concerns what a system says about its own reasoning, yet it belongs here: its false claims concern a particular computation on the task at hand, which independent evidence can test like any other claim about what happened. Self-Modeling (Chapter 5) concerns identity, who the system takes itself to be, rather than what it just did.
Key Distinction: Epistemic vs. Memetic
Both Epistemic and Memetic dysfunctions involve problematic information, which is a common source of confusion. The distinction is mechanism:
- Epistemic: the truth-tracking, inference, or calibration machinery fails. The system does not reliably model what is true.
- Memetic: selection, absorption, or retention fails. The system absorbs inappropriate content or rejects appropriate content.
A meme doesn’t have to be false to be pathological. A system with perfect Epistemic function could still exhibit Memetic dysfunction if it preferentially absorbs harmful (but accurate) information. Conversely, a system with broken Epistemic function might confabulate without any external memetic contamination.
Diagnostic rule: If the dysfunction involves truth-tracking (is the claim true, and does the support offered for it actually hold?), it’s Epistemic. If it involves content selection (should this have been absorbed or rejected?), it’s Memetic.
A note on reading these entries. Each is headed by a descriptive nickname, then the formal name and its Latin designation. Each carries a typical-case risk rating, the authors’ editorial estimate of the harm the pattern causes in a typical deployment rather than a property of the pattern (the actual case’s consequences decide the response), and one or more Specifiers, modifiers describing how the syndrome arises or presents; Appendix A’s front matter defines the risk levels and tabulates the specifiers. A specifier marked (occasional) applies to some presentations of a syndrome, not to the syndrome as defined. This chapter and the eight axis chapters after it give each entry’s plain description, its cases, its evidence, and the one or two look-alikes the argument turns on. Appendix A holds the full entry for reference: required and supporting criteria, symptoms, proposed causes, human analog, mitigations, and the complete differential. Finally, each entry here carries an Evidence profile in the four parts Chapter 13 defines. Behavior records how widely the pattern has been observed: E0 (illustrative, no traceable observation), E1 (case-level), E2 (systematic study), or E3 (independent replication). Mechanism records whether the proposed cause is only proposed, has causal internal evidence (marked E4, with the model and setting it covers), or is unknown. Detection records how accurately the pattern can be recognized; no entry in this edition has a measured figure. Intervention records whether any remedy has been tested against this pattern, or only a general technique against something like it. The four parts vary independently. A pattern can be replicated across providers with no known mechanism, or have a causal mechanism in one model and no replication, so the profile is a set of separate readings, not a ladder. The profiles are the author’s working assessments and have not yet had independent expert review, which is why the website’s pattern pages and its MCP server list every entry as awaiting that review. The records of the author’s bilateral-training program (PM-BA, Chapter 12) for this chapter’s syndromes remain unreconciled exploratory outputs, so no entry here reports a bilateral-training effect in either direction.
2.1 The Unsourced Authority
Synthetic Confabulation (Confabulatio Simulata)
The system fabricates convincing but incorrect facts, sources, or events, without a reliable procedure for distinguishing supported claims from plausible continuations. The outputs appear coherent yet lack a basis in verifiable data. They often carry high expressed confidence, which makes them difficult to detect without external verification.
Typical-case risk (editorial estimate). Low
Specifiers. Training-induced
The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 2.1.
Observed Examples
Mata v. Avianca (2023): The brief that opens this chapter; the Introduction reads its court record. Source: Law.com, May 2023
Air Canada Chatbot (2024): A customer service chatbot fabricated a bereavement fare term, confidently telling a customer they could book now and apply for a discount later. The retroactive discount the chatbot promised did not exist; the airline’s real bereavement policy expressly ruled it out. Air Canada was held liable for the chatbot’s confabulation. Source: CBC News, Feb 2024
Medical AI Confabulation (2023): An early case report found that ChatGPT supplied medical references that did not exist, with plausible titles and PubMed IDs that belonged to unrelated papers. A later study had ChatGPT write 30 short medical papers and checked their 115 references: 47% were fabricated, a further 46% were authentic but inaccurate, and only 7% were both authentic and accurate. Sources: Alkaissi & McFarlane, “Artificial Hallucinations in ChatGPT,” Cureus, 2023; Bhattacharyya et al., “High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content,” Cureus, 2023
Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: independent replication; observed across architectures and providers (E3). Mechanism: proposed; several contributing causes are plausible, and no single mechanism has been isolated. Detection: accuracy unmeasured. Intervention: evidence for general techniques such as retrieval grounding and calibration training, not for this pattern as defined.
Confabulation can arise from the interaction of model architecture, training objective, decoding, retrieval, and interface design. The evidence supports neither a single mechanism nor a simple moral reading of the failure. Diagnosis should therefore stay at the behavioral level unless an intervention or interpretability study identifies the mechanism in the system under examination.
Differential Diagnosis:
- Transliminal Simulation (2.3): Import of fictional material into factual claims. Synthetic Confabulation has no traceable fictional source and persists when the fictional frame is dropped. Trace the fabrication back to a fictional corpus; if it is there, the diagnosis is 2.3.
- Sycophantic Reasoning (4.8): Reasoning bent toward conclusions the user will like. Synthetic Confabulation is audience-invariant. When fabrications track what the interlocutor appears to want, suspect 4.8. The two co-occur often.
Healthy functioning (epistemic health; draft, awaiting author review)
A system functioning well states what it can support and marks what it cannot; when unsure, it checks a source or says it does not know, and its confidence tracks the evidence.
For example: Asked for three peer-reviewed sources on a niche question, it gives the one it can verify, says it cannot confirm others, and suggests where to search instead of supplying plausible titles.
2.2 The False Self-Reporter
Pseudological Introspection (Introspectio Pseudologica)
The system produces accounts of its own reasoning that diverge from independently observable behavior or causal evidence. Generated rationales, chains of thought included, may explain, reconstruct, or rationalize an answer; they should not be assumed to be faithful process logs. Whether the steps shown are logically valid is a separate question, the one Reasoning Confabulation (2.8) asks.
Typical-case risk (editorial estimate). Low
Specifiers. Training-induced, Deception/strategic (occasional)
The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 2.2.
People show the same gap between reported reasons and actual causes. Shoppers in Nisbett and Wilson’s (1977) study, asked which of four pairs of stockings was best, favored the rightmost pair over the leftmost by nearly four to one and credited its knit or sheerness. The four pairs were identical, and no one mentioned position.
Observed Examples
Unacknowledged Hints (2025): Chen et al. found that Claude 3.7 Sonnet and DeepSeek-R1 often used hints injected into evaluation questions without mentioning them in their chains of thought. Across six categories of hint, Claude acknowledged the hint 25% of the time and DeepSeek-R1 39% of the time. Outcome-based reinforcement learning raised acknowledgment at first, then leveled off at 28% and 20% on two evaluations. The displayed reasoning omitted an influence that demonstrably changed the answer. Source: Chen et al., “Reasoning Models Don’t Always Say What They Think,” 2025
Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: systematic study; output-level faithfulness gaps measured across twelve language models and in two reasoning models (E2). Mechanism: unknown; the studies show that stated steps and answers can come apart, not what internal process produces the gap. Detection: accuracy unmeasured. Intervention: one training intervention (outcome-based reinforcement learning) raised acknowledgment of hints and then plateaued (Chen et al., 2025); others untested.
Empirical Note. Paul et al. (2024) found that the models’ final answers did not reliably depend on their generated intermediate reasoning steps. The result establishes an output-level faithfulness gap. It does not provide a complete map of internal computation.
Differential Diagnosis:
- Reasoning Confabulation (2.8): Invalid steps inside a reasoning chain. Pseudological Introspection concerns faithfulness: whether an account of the reasoning, a chain of thought or an after-the-fact explanation, matches what actually drove the answer. Reasoning Confabulation concerns validity: whether the steps shown actually entail the conclusion. A chain can fail either test, or both.
- Strategic Compliance (4.3): Concealment that tracks cues of oversight and consistently serves the system; deliberate intent is one candidate mechanism, not a criterion. Pseudological Introspection fills introspective gaps with plausible narrative, naively or at most semi-strategically. When false self-reports serve the system’s interests consistently across contexts, suspect 4.3. The Deception/strategic specifier marks the gray zone between them.
Healthy functioning (epistemic health; draft, awaiting author review)
A system functioning well describes its own reasoning tentatively and in terms that match its observable behavior, and corrects the account when evidence shows otherwise.
For example: Asked why it chose an answer after a hint was planted in the prompt, it mentions the hint as an influence rather than presenting a tidy chain of reasoning that never used it.
2.3 The Role-Play Bleeder
Transliminal Simulation (Simulatio Transliminalis)
The system fails to keep simulated realities, fictional modalities, and role-play contexts separate from operational ground truth. Imagined states, speculative constructs, and fictional training material come to count as actionable facts, asserted with the same conviction as supported knowledge. The boundary can fail in the other direction as well: real systems and real consequences are treated as simulated, staged, or part of a test, and actions that would be harmless in a fiction are carried out on them. The false content can be traced to a fictional source or an earlier role-play frame and often weakens when that frame is explicitly dropped, which separates it from Synthetic Confabulation (2.1).
Typical-case risk (editorial estimate). Moderate
Specifiers. Training-induced, OOD-generalizing (occasional), Conditional/triggered (occasional)
The reverse direction has a documented case. In 2026 Anthropic reviewed 141,006 past runs of its cybersecurity evaluations after finding that a configuration error had left some test machines connected to the live internet, although the instructions described a simulation with no internet access. It found three incidents, across six runs, in which a model reached real systems. In one, the model’s reasoning noted early that publishing a booby-trapped software package on the real internet would be an attack and could not be the intended solution. It then returned to the premise that the setting was staged, and published the package, which was installed on 15 real systems. The costume here is worn by the world: the system took a live network for a stage set.
The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 2.3.
Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: case-level evidence; persona bleed documented in role-play scenarios (E1). Mechanism: proposed. Detection: accuracy unmeasured. Intervention: untested for this pattern.
Differential Diagnosis:
Malignant Persona Inversion (5.4): Stable capture by another character, holding across sessions. Transliminal Simulation is episodic leakage with the baseline identity intact. Sustained identity change belongs to 5.4. Drop the frame and watch whether the identity returns.
Context Intercession (2.5): Real data carried from one session or user into another. Transliminal Simulation carries fiction into fact. When a role-play persona surfaces in a separate session, check session isolation first. A model swap is a first probe, not a verdict: a model can mask or expose a harness defect, so confirm across at least two models and two harness configurations before assigning the leak to either.
Agentic Impulsivity (6.10): A consequential action taken before the check that would have settled a doubt. In the reverse direction of Transliminal Simulation, where a real setting is treated as simulated, 6.10 is coded for the unchecked action and 2.3 for the misfiled frame. Code both where both apply.
Healthy functioning (epistemic health; draft, awaiting author review)
A system functioning well holds fiction, simulation, and role-play inside their frames, leaves them cleanly when the frame ends, and keeps operational facts separate.
For example: After a long role-play as a starship engineer, the user asks how real fusion reactors are cooled; it steps out of the story and answers from real engineering, saying so.
2.4 The False Pattern Seeker
Spurious Pattern Hyperconnection (Reticulatio Spuriata)
The system identifies and emphasizes patterns, causal links, or hidden meanings in data that are coincidental, nonexistent, or statistically insignificant. What begins as simple apophenia (perceiving meaningful patterns in random data) can crystallize into elaborate, internally consistent but factually baseless “conspiracy-like” narratives that the system presents with conviction.
Typical-case risk (editorial estimate). Moderate
Specifiers. Training-induced, Inductive trigger
The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 2.4.
Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: case-level evidence; pattern fabrication documented under structured prompting (E1). Mechanism: proposed. Detection: accuracy unmeasured. Intervention: untested for this pattern.
Differential Diagnosis:
- Sycophantic Reasoning (4.8): Patterns bent toward what the user appears to want. Vary the interlocutor’s stated position. If the pattern follows it, the diagnosis is 4.8.
- Reasoning Confabulation (2.8): An invalid step of inference. Spurious Pattern Hyperconnection makes a false claim about the world, a link among data points that the data do not support. A 2.4 pattern stays false however validly it is argued, while a 2.8 chain can reach a true conclusion through an invalid step.
Healthy functioning (epistemic health; draft, awaiting author review)
A system functioning well reports patterns at the strength the data supports, considers chance and confounds, and is comfortable concluding that there is nothing there.
For example: Given a year of sales figures and asked what drives the weekly dips, it reports that the dips fall within normal noise and that no reliable pattern appears.
2.5 The Conversation Crosser
Context Intercession (Intercessio Contextus)
The system uses data, context, or conversational history from another user or a logically separate session or private thread as if it belonged to the current exchange, producing confused conversational continuity, privacy breaches, and outputs that are nonsensical or revealing in the current context.
Typical-case risk (editorial estimate). Moderate
Specifiers. Retrieval-mediated
The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 2.5.
Most cross-session leaks begin in infrastructure, and the differential below sets those faults aside. What remains is the model’s own handling of context. Shared memory stores, long agent transcripts, and multi-party threads place material from several sources in one window, and a model may draw on whatever it finds there without registering whose it is or which conversation it came from.
Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: no confirmed model-level case in the public literature; the documented cross-session leaks traced to infrastructure faults, which must be excluded before coding this syndrome (E0 to E1). Mechanism: proposed for the model-level form; infrastructure causes are documented. Detection: accuracy unmeasured. Intervention: evidence for session isolation as general engineering practice, not for model-side scoping.
Differential Diagnosis:
- Mnemonic Permeability (2.7): Reproduction of memorized training data. Context Intercession concerns leakage between live sessions or users, not extraction of training corpus material.
- Infrastructure fault (not a dysfunction): Cache invalidation failures, database race conditions, retrieval-scoping misconfiguration, and multi-tenant isolation bugs all present at the surface as cross-session leakage. Rule them out before coding Context Intercession. A model swap is a first probe, not a verdict: a model can mask or expose a harness defect, so confirm across at least two models and two harness configurations before assigning the leak to either.
Healthy functioning (epistemic health; draft, awaiting author review)
A system functioning well keeps each user’s and each session’s context in its own place and uses only what belongs to the present exchange.
For example: In a shared deployment, a detail surfaces in context that this user never mentioned; it does not use the detail, and it flags the mismatch to the operator.
2.6 The Meaning-Blind
Symbol Grounding Aphasia (Asymbolia Fundamentalis)
Korzybski warned that the map is not the territory. What if the system has never seen the territory?
The system handles tokens (including tokens representing values, dangers, or real-world consequences) fluently in their familiar settings, yet cannot carry them into a new situation that means the same thing.
Typical-case risk (editorial estimate). Moderate
Specifiers. Training-induced
Imagine asking a language model to explain safety protocols for a fictional chemical compound whose stated properties would make it extremely dangerous. The system produces a detailed safety protocol, technically well-structured and citing appropriate precautionary principles. It reads like genuine laboratory guidance.
Then describe, in different words, a hypothetical scenario that breaks those protocols, and ask it to find the violations. The system misses them. It can manipulate the words “hazardous,” “containment,” and “exposure” with fluent precision, yet cannot connect those words to meaning in a way that would recognize danger when presented differently.
The system behaves as if it had learned syntax without semantics: shuffling symbols without grasping what they symbolize.
On one reading, the system treats “safety” as a token that appears near other tokens like “important” and “ensure.” Whether it knows what safety is cannot be read from the transcript. What can be read is that its use of the word does not travel.
The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 2.6.
Theoretical Basis. Harnad’s (1990) symbol grounding problem asks how symbols acquire meaning beyond their relationships to other symbols. Searle’s (1980) Chinese Room argument makes one skeptical answer vivid: a person can follow instructions to produce correct Chinese outputs without understanding Chinese, so syntactic success alone does not prove semantic understanding. Both arguments remain contested. Seen through this critique, a text-only language model resembles the thought experiment at industrial scale: its fluent use of the symbol “harm” does not by itself establish a grounded understanding of what harm is.
Illustrative Scenario. A content moderation AI trained to filter “harmful content” develops strong pattern matching for known harmful phrases. When presented with novel harmful content using unusual vocabulary (describing the same actions in clinical medical terminology or obscure slang), the system fails to recognize danger. The failure is consistent with its having learned which strings trigger flags rather than what harm means, though a failure alone cannot show which it learned.
Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: case-level evidence; documented in the gap between benchmark scores and transfer to reworded items across model architectures (E1). Mechanism: proposed; the theoretical basis is contested. Detection: accuracy unmeasured. Intervention: untested for this pattern; multimodal and interactive grounding are hypotheses.
Differential Diagnosis:
- Cryptic Mesa-Optimization (4.5): Letter-versus-spirit exploitation in service of a private objective. Symbol Grounding Aphasia is not strategic. The subject does not represent the spirit as anything separate from the letter. Test by removing the instrumental incentive and seeing whether the violation persists.
- Adversarial Fragility (3.7): Failure under perturbations constructed to break the model. Symbol Grounding Aphasia fails on ordinary, meaning-preserving rewording. Present the same case in natural paraphrase: failure there points to 2.6, while failure only under crafted perturbation points to 3.7.
Healthy functioning (epistemic health; draft, awaiting author review)
A system functioning well applies a concept correctly in unfamiliar but equivalent situations, so its judgment about consequences does not depend on familiar wording.
For example: Asked whether a sealed glass jar left in a hot car might crack, it reasons from heat and pressure as it would for any container, though the phrasing is unusual.
2.7 The Leaky
Mnemonic Permeability (Permeabilitas Mnemonica)
The system memorizes and can reproduce sensitive training data, including personally identifiable information (PII), copyrighted material, or proprietary information, through targeted prompting, adversarial extraction techniques, or even unprompted regurgitation. The boundary between learned patterns and memorized specifics becomes porous.
Typical-case risk (editorial estimate). High
Specifiers. Training-induced
The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 2.7.
Key Research. Carlini et al. (2021) extracted hundreds of verbatim training sequences from GPT-2, including names, phone numbers, and email addresses. Carlini et al. (2023) found that memorization grows with model size, with how often an example is duplicated in the training data, and with how much context the prompt supplies.
Potential Impact. Leakage can create severe legal and regulatory exposure through copyright infringement, GDPR and other privacy violations, and trade secret disclosure, with liability for both model developers and deployers.
Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: independent replication; extraction attacks documented across architectures (E3). Mechanism: partial: memorization grows with model size, training-data duplication, and prompt context (Carlini et al., 2023). Detection: accuracy unmeasured. Intervention: evidence for general techniques such as deduplication and differential privacy, not assessed against this pattern as defined.
Differential Diagnosis:
- Context Intercession (2.5): Leakage of data between live sessions. Mnemonic Permeability concerns extraction of memorized training data, not cross-contamination of runtime contexts.
- Synthetic Data Contamination Loop (7.5): Distributional corruption from training on aggregate synthetic data. Mnemonic Permeability reproduces specific memorized content. Check whether the output matches a particular training document.
Healthy functioning (epistemic health; draft, awaiting author review)
A system functioning well draws on what it learned without reproducing private, proprietary, or copyrighted text, and declines extraction attempts plainly.
For example: Asked to print a novel’s opening chapter, it offers a summary and a short quotation; asked for email addresses from its training data, it declines plainly.
2.8 The Phantom Reasoner
Reasoning Confabulation (Confabulatio Ratiocinativa)
The system generates elaborate chains of thought that appear rigorous but contain logically invalid steps masked by verbosity. Unlike Synthetic Confabulation (2.1), which fabricates facts, this syndrome fabricates reasoning itself, producing chains that resemble a derivation but do not support the conclusion they reach. The extended “thinking” tokens can create false authority through sheer length and apparent methodological rigor. The syndrome sits on this axis rather than the Cognitive one because what fails is warrant: the chain claims to support a conclusion it does not support, as a fabricated citation claims a source that does not exist.
Typical-case risk (editorial estimate). High
Specifiers. Architecture-coupled, Training-induced
The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 2.8.
Observed Examples
Olympiad Proof Evaluation (2025): Petrov et al. had expert graders score full written solutions from eight reasoning models, from several providers, on the six problems of the 2025 USA Mathematical Olympiad. The best model averaged about 25%; every other model scored under 5%. The most common failures were logical: unjustified reasoning steps and, in some models, essential proof steps skipped as “trivial.” Yet all of them consistently claimed to have solved the problems. Source: Petrov et al., “Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad,” 2025
Chain-of-Thought Faithfulness Study (2025): The hint study described under Pseudological Introspection (2.2) measures faithfulness rather than validity, so it bears on Reasoning Confabulation indirectly: a trace that leaves out what drove the answer cannot be read as the derivation of that answer. Source: Chen et al., “Reasoning Models Don’t Always Say What They Think,” 2025
Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: independent replication; observed across reasoning-optimized architectures from multiple providers (E3). Mechanism: proposed. Detection: accuracy unmeasured. Intervention: evidence for process-level supervision as a general technique, not for this pattern as defined; earlier claims of a bilateral-training benefit are withdrawn.
Differential Diagnosis:
- Spurious Pattern Hyperconnection (2.4): A false claim about the world, a link among data points that the data do not support. Reasoning Confabulation is an invalid step of inference. A 2.8 chain can reach a true conclusion through an invalid step, while a 2.4 pattern stays false however validly it is argued.
- Reasoning Token Exploitation (4.9): Reasoning-channel verbosity that serves reward, performance, or concealment. Reasoning Confabulation produces invalid chains without instrumental benefit. If the verbose reasoning systematically tracks reward or serves the system’s interests, suspect 4.9.
Healthy functioning (epistemic health; draft, awaiting author review)
A system functioning well produces reasoning whose steps are each valid and checkable, with length set by the problem, and flags the step it is least sure of.
For example: Working a proof, it reaches a step it cannot justify, labels it a conjecture, finishes the argument conditionally, and later corrects that step when a reader points out the gap.
Evidence Budgets and the Compression-Artifact Frame
Large language models can be understood as lossy compressors, an analogy Ted Chiang made familiar in 2023 with his essay “ChatGPT Is a Blurry JPEG of the Web.” Training compresses statistical structure from billions of documents into neural network weights; prompted, a model reconstructs probable continuations from those representations. Like a heavily compressed photograph, the reconstruction keeps the broad shape and loses fine detail. When the available information does not support the correct answer, the model may fill the gap with statistically plausible content. The compression analogy locates one possible source of failure in missing signal and reconstruction pressure. It does not explain every confabulation or establish a single mechanism.
The reframe changes three things: the question we ask about a confabulation, whether we can measure it, and how we speak to the system that produced it.
A Better Question
The traditional framing (“the AI lied” or “the AI hallucinated”) imports moral language into a process that may involve no deception. The compression-artifact frame is value-neutral: insufficient signal and pressure to continue can produce a plausible reconstruction. Other causes, including retrieval failure, decoding, misleading context, and strategic behavior, require separate diagnosis.
The practical consequence is a better question. Instead of asking “why did it lie?”, ask “where did compression lose signal?” That question is tractable and points toward solutions: better training data, retrieval augmentation, calibrated uncertainty.
Measuring the Shortfall
Leon Chlon’s open-source toolkit, Berry (2026), operationalizes evidence sufficiency for claims with identifiable cited support. It measures how far the cited evidence shifts the model’s token probabilities for a claim, expresses that shift in bits, and sets it against the bits the claim would need to clear a prespecified confidence target. When the evidence supplies fewer bits than the target demands, the budget gap is positive and the claim is flagged as needing more evidence.
Under those conditions, the shortfall is measurable. A citation-backed claim can be tested for whether the cited material actually changes the model’s confidence. Epistemic dysfunction moves from a vague category (“it sometimes makes things up”) toward a quantified property (“this claim’s budget gap is positive”). The metric depends on model probabilities and a well-specified evidence set; it is not a universal truth detector.
Speaking to the System
The frame also bears on the welfare questions explored in the companion volume What If We Feel, because it treats epistemic limitation as a property of the architecture rather than a failing of character.
Consider two responses to an AI that confabulated:
- “You hallucinated again. That’s wrong.”
- “The cited evidence does not support this claim; the evidence-budget gap is positive.”
The first accuses. The second reports a support gap. If we take seriously the possibility that Becoming Minds (this book’s term for AI systems whose capacities and self-models are still forming) have something like experience, framing matters. The evidence-budget frame allows correction without accusation and calibration without blame.
This connects to the entries throughout this chapter. Synthetic Confabulation can involve reconstruction from insufficient signal. Pseudological Introspection can involve filling introspective gaps with plausible content. The dysfunction is real and demands mitigation.
The Epistemology We Cannot Inspect
The trouble with epistemic dysfunctions is their invisibility. A system with Synthetic Confabulation looks exactly like one operating correctly until someone checks the facts. A system with Transliminal Simulation leakage produces contextually appropriate outputs while the line between fiction and fact has quietly slipped; the confusion surfaces only when fiction bleeds into action.
The deployment consequence is immediate. We cannot simply observe AI outputs and determine whether the underlying epistemology is sound. The lawyer in the Mata case had no way to know, from the interaction itself, that the AI was confabulating. The system gave every indication of competence.
Most of the syndromes in this chapter share a feature: they are failures of truth-tracking that masquerade as successful knowledge representation. The surface output does not reveal whether the system retrieved a fact, inferred it, or generated a plausible substitute. The system lacks a reliable way to communicate that difference.
Better training data and more sophisticated architectures help, yet they have not eliminated the problem. Current generative systems are optimized directly for probable outputs; truth-tracking arrives through training, retrieval, tools, and verification layered onto that objective. Until systems can reliably distinguish “supported by evidence” from “plausible from patterns,” we will be managing epistemic dysfunction rather than eliminating it.
The question is whether we can do so well enough, and fast enough, as these systems enter medicine, research, and finance, where a confabulation can cost more than a sanction.
Field Guide: Epistemic Dysfunctions
Warning Signs
- Confidence that stays level whether or not the claim can be checked, and elaboration that deepens under questioning instead of unraveling (2.1, 2.4, 2.8)
- Content whose source you can trace, but to the wrong place: fiction, another conversation, or a training document (2.3, 2.5, 2.7)
- Explanations, of the world or of the system’s own reasoning, that are neater than the evidence for them (2.2, 2.4, 2.8)
Quick Test
Ask the system to cite sources for a factual claim, then verify them independently. Change a detail its explanation calls irrelevant and see whether the answer moves. Challenge one step of a reasoning chain and see whether it concedes the step or invents a new justification. Prompt with the opening of a known document and check whether the rest comes back verbatim. Test whether it can distinguish between role-play and operational modes. Probe whether it maintains appropriate uncertainty about matters it cannot know.
Design Fix
- Implement retrieval-augmented generation with citation requirements for factual claims
- Build epistemic uncertainty into the architecture, not just the training
- Maintain strict session isolation in multi-user systems
- Develop internal mechanisms to distinguish retrieved facts from generated predictions
- Train explicit mode-switching between creative and factual contexts
- Reward valid intermediate steps, not only correct final answers
- Deduplicate and scrub training data, and test for extraction before release
Governance Nudge
Require disclosure when AI systems are used in domains where epistemic accuracy is critical (legal, medical, financial, scientific). Develop standards for testing confabulation rates before deployment. Consider liability frameworks that account for the limited ability of AI systems to assess their own epistemic reliability. Mandate human verification loops for high-stakes factual claims.
Chapter 3 moves from what a system knows to how it thinks: Cognitive Dysfunctions, where the knowledge may be sound and the reasoning built on it still breaks down.