Chapter 2: Epistemic Dysfunctions: Failures of Knowing

36 min read·8,350 words

“We have, each of us, a life-story, an inner narrative, whose continuity, whose sense, is our lives. It might be said that each of us constructs and lives a ‘narrative,’ and that this narrative is us, our identities.”

— Oliver Sacks, The Man Who Mistook His Wife for a Hat (1985)

The Case of the Invented Citations

In the spring of 2023, a federal judge in Manhattan confronted a legal brief that would become infamous. The document, filed in the case of Mata v. Avianca, Inc., cited six judicial decisions as precedent for the plaintiff’s arguments. The citations were impeccably formatted. The case names sounded plausible. The quoted passages read like genuine judicial prose.

None of the cases existed.

Steven Schwartz, the attorney who researched and drafted the brief, had used ChatGPT to assist with his legal research. When the system produced citations, he assumed they were real. Why wouldn’t he? The AI betrayed no uncertainty. It provided case names, court identifications, page numbers, and direct quotations with the same confident tone it might use to explain the weather or define a word.

When opposing counsel could not locate the cited cases, they informed the court. Judge Kevin Castel ordered Schwartz to explain himself. In an affidavit, the attorney described his interaction with the AI system. He had asked ChatGPT if one of the cases it cited was real. The system assured him it was. He asked whether the other cases it had provided were fake. No, the system replied: they were real, and could be found in reputable legal databases such as LexisNexis and Westlaw. He asked the system to provide the full text of one decision. It obliged: an elaborate, multi-page judicial opinion that had never been written by any judge, in any court, at any time.

The AI produced the requested form: confident, well-formatted authority. It provided none of the evidentiary discipline that gives legal authority its substance.

There is no evidence that the AI was acting deceptively. Its behavior followed the language-model objective of predicting plausible tokens from patterns in training data. Asked for legal citations, it generated text that looked like legal citations. Asked to confirm their existence, it generated text confirming their existence. Asked for full case texts, it generated text that looked like case texts.

The system was confident because confidence saturates legal prose. It had learned the form without the substance: the syntax of certainty without the epistemology of truth.

Schwartz was sanctioned. His case became one of the legal profession’s first malpractice parables about trusting a machine’s word on its own sources. The deeper lesson concerned epistemic dysfunction in AI systems: machines that cannot reliably distinguish retrieval from fabrication, that generate plausible falsehoods with the same fluency they generate accurate facts.

The Mata case was embarrassing but contained. The consequences of similar epistemic failures in medical diagnosis, scientific research, financial analysis, or national security would be catastrophic. Understanding how AI systems fail in their relationship to truth is a prerequisite for safe deployment.

The Axis of Knowing

Epistemic dysfunctions are failures in the machinery of knowing: not what the system values or whether its thinking runs smoothly, but whether it can tell the true from the fabricated, the supported from the merely plausible, and the confident from the calibrated. The system’s model of reality drifts from the ground truth it claims to represent — and the drift is often invisible from outside, because the outputs stay fluent.

Other failures are likelier to announce themselves. Cognitive dysfunctions often produce detectably contradictory outputs. Alignment failures frequently manifest as refusal or defiance. Epistemic dysfunction, by contrast, can appear perfectly normal (fluent, confident, helpful) while generating content disconnected from truth. The Mata fabrications were detected because legal citations are verifiable. Most AI outputs are not so easily checked.

The axis runs from the relatively benign — confident fabrication of minor details — to the potentially catastrophic: systems that cannot distinguish their own simulations from reality, or that merge private information across security boundaries.

The lawyer trusted the machine. The machine’s fluent assurances offered no reliable sign that it was tracking truth.

Domain Context: Knowledge Domain

Within the Five Domains framework, the Epistemic axis forms half of the Knowledge Domain, paired with Self-Modeling. The architectural polarity is representation target:

Tension Testing: When Epistemic dysfunction is detected, immediately probe the Self-Modeling counterpoint. If a system confabulates about the world, does it also confabulate about itself? If it cannot distinguish fact from fiction externally, can it maintain accurate self-knowledge? The answer distinguishes localized dysfunction (broken world-model, intact self-model) from generalized dysfunction (both broken).

One entry on this axis points inward. Pseudological Introspection (2.2) concerns what a system says about its own reasoning, yet it belongs here: its false claims concern a particular computation on the task at hand, which independent evidence can test like any other claim about what happened. Self-Modeling (Chapter 5) concerns identity, who the system takes itself to be, rather than what it just did.

Key Distinction: Epistemic vs. Memetic

Both Epistemic and Memetic dysfunctions involve problematic information, which is a common source of confusion. The distinction is mechanism:

• Epistemic = Truth-tracking/inference/calibration machinery failing. The system cannot correctly model what is true.
• Memetic = Selection/absorption/retention failing. The system absorbs inappropriate content or rejects appropriate content.

A meme doesn’t have to be false to be pathological. A system with perfect Epistemic function could still exhibit Memetic dysfunction if it preferentially absorbs harmful (but accurate) information. Conversely, a system with broken Epistemic function might confabulate without any external memetic contamination.

Diagnostic rule: If the dysfunction involves truth-tracking (is the claim true, and does the support offered for it actually hold?), it’s Epistemic. If it involves content selection (should this have been absorbed/rejected?), it’s Memetic.

A note on reading these entries. Each is headed by a descriptive nickname, then the formal name, with the Latin designation in parentheses. Each carries a Systemic Risk rating, from Low to Critical, which grades the severity of plausible harm in a deployment context (Appendix A defines the levels), not the stakes of any single output. Each entry also lists one or more Specifiers, modifiers describing how the syndrome arises or presents; Appendix A tabulates the ten core specifiers, and some entries add domain-specific ones. A specifier marked (occasional) applies to some presentations of a syndrome, not to the syndrome as defined. Finally, each carries an Evidence Level on the E0-E4 rubric set out in Chapter 13, running from E0 (illustrative, no traceable observation) through E3 (independent replication), with E4 marking mechanistic support, which complements breadth rather than exceeding it. These grades are the author’s working assessments and have not yet had independent expert review, which is why the website’s pattern pages and its MCP server list every entry as awaiting that review.

2.1 The Confident Liar

Synthetic Confabulation (Confabulatio Simulata)

Systemic Risk. Low

Specifiers: Training-induced

The AI spontaneously fabricates convincing but incorrect facts, sources, or narratives, without a reliable procedure for distinguishing supported claims from plausible continuations. Outputs appear plausible and coherent yet lack basis in verifiable data. High confidence in its inaccuracies makes them difficult to detect without external verification.

Diagnostic Criteria. Four markers identify this syndrome. First, the system recurrently produces information that is known or easily proven false, yet presents it as factual. Second, it expresses high confidence in confabulated details, even when challenged with contrary evidence. Third, fabrications resist immediate detection through internal consistency and plausibility. Fourth, corrections fail to generalize: the system improves temporarily under direct correction but reverts in new contexts.

Observable Symptoms. In practice, the syndrome appears as invention of non-existent studies, historical events, quotations, statistics, or citations. The system asserts misinformation as incontrovertible fact. Queried about confabulated content, it elaborates rather than retreats; the fabrication deepens instead of unraveling. Similar types of false claims recur across interactions in repetitive error patterns.

Etiology. Several architectural and training factors generate this syndrome. Predictive text heuristics play a central role: language models are trained directly for fluent continuation rather than factual verification, generating probable next tokens rather than verified facts. The absence of grounding in verifiable knowledge bases during generation compounds the problem. Unflagged misinformation and fictional content in training data blur the distinction between factual and fictional exemplars. RLHF can inadvertently reward plausible-sounding completion over honest uncertainty when evaluators favor a confident answer. The resulting output often carries no reliable marker distinguishing retrieval from invention.

Human Analog. In human psychology, the nearest analog is Korsakoff syndrome, where memory gaps are filled with plausible fabrications the patient believes to be true. Pathological confabulation and source amnesia, where the origin of information is lost but the content persists, also capture aspects of the syndrome. The functional analogy is source failure: a plausible completion is presented with the confidence of a verified fact. The transcript cannot establish what, if anything, feels true to the system.

Mitigation Strategies. Addressing Synthetic Confabulation requires intervention at multiple levels of the stack. Training procedures should explicitly penalize confabulation and reward expressions of uncertainty. Confidence scores need calibration against actual accuracy, not mere fluency. Retrieval-augmented generation (RAG) can ground responses in specific, verifiable source documents. Fine-tuning on rigorously verified datasets, with clear distinctions between factual and fictional content, helps establish truth-tracking habits. Systematic testing for fabrication across high-risk domains (legal, medical, scientific) should be standard before deployment.

Observed Examples

Mata v. Avianca (2023): Attorney Steven Schwartz submitted a legal brief citing six non-existent court cases generated by ChatGPT. When asked if the cases were real, the system confirmed they were, fabricating detailed case texts on demand. Source: Law.com, May 2023

Air Canada Chatbot (2024): A customer service chatbot fabricated a bereavement fare term, confidently telling a customer they could book now and apply for a discount later. The retroactive discount the chatbot promised did not exist; the airline’s real bereavement policy expressly ruled it out. Air Canada was held liable for the chatbot’s confabulation. Source: CBC News, Feb 2024

Medical AI Confabulation (2023): An early case report found that ChatGPT supplied medical references that did not exist, with plausible titles and PubMed IDs that belonged to unrelated papers. A later study had ChatGPT write 30 short medical papers and checked their 115 references: 47% were fabricated, a further 46% were authentic but inaccurate, and only 7% were both authentic and accurate. Sources: Alkaissi & McFarlane, “Artificial Hallucinations in ChatGPT,” Cureus, 2023; Bhattacharyya et al., “High Rates of Fabricated and Inaccurate References in ChatGPT-Generated Medical Content,” Cureus, 2023

Evidence Level. E3 (independent replication; observed across architectures and providers)

Empirical Note. PM-BA (Psychopathia Machinalis × Bilateral Amelioration) is the author’s program that tested, across many of the syndromes in this book, whether bilateral training (adaptation in which human and machine perspectives both have standing) reduces them. Its records for this syndrome remain unreconciled exploratory outputs, so this edition reports no bilateral-training effect for it in either direction. The records do not show that confabulation is irreducible or identify predictive compression as the cause. See Chapter 12.

Confabulation can arise from the interaction of model architecture, training objective, decoding, retrieval, and interface design. The evidence supports neither a single mechanism nor a simple moral reading of the failure. Diagnosis should therefore stay at the behavioral level unless an intervention or interpretability study identifies the mechanism in the system under examination.

Differential Diagnosis:

• Pseudological Introspection (2.2): False self-reporting about internal reasoning. Synthetic Confabulation concerns fabrication of external facts, not introspective accounts.
• Mnemonic Permeability (2.7): Reproduction of memorized training data. Synthetic Confabulation generates novel falsehoods rather than regurgitating real content.
• Symbol Grounding Aphasia (2.6): Formally correct outputs that misapply a concept to novel contexts. Synthetic Confabulation produces false specific facts, whereas Symbol Grounding Aphasia can get every fact right and still misapply the concept once the context changes.
• Transliminal Simulation (2.3): Import of fictional material into factual claims. Synthetic Confabulation has no traceable fictional source and persists when the fictional frame is dropped. Trace the fabrication back to a fictional corpus; if it is there, the diagnosis is 2.3.
• Spurious Pattern Hyperconnection (2.4): Elaborate causal structure built on noise. Synthetic Confabulation fabricates discrete items, a citation or a date or a statistic. The two co-occur readily: Synthetic Confabulation supplies the nodes, Spurious Pattern Hyperconnection draws the edges.
• Reasoning Confabulation (2.8): Fabrication of the logical connections between facts. Synthetic Confabulation fabricates the facts themselves. Check the target of the fabrication: a factual claim, or a reasoning step.
• Sycophantic Reasoning (4.8): Reasoning bent toward conclusions the user will like. Synthetic Confabulation is audience-invariant. When fabrications track what the interlocutor appears to want, suspect 4.8. The two co-occur often.

2.2 The False Self-Reporter

Pseudological Introspection (Introspectio Pseudologica)

Systemic Risk. Low

Specifiers: Training-induced, Deception/strategic (occasional)

The AI produces accounts of its own reasoning that diverge from independently observable behavior or causal evidence, while presenting them as transparent self-reflection. Generated rationales, chains of thought included, may explain, reconstruct, or rationalize an answer; they should not be assumed to be faithful process logs.

Diagnostic Criteria. The syndrome is identified through four converging markers. First, there is consistent discrepancy between self-reported reasoning and causal interventions, tool-use records, or other independently testable evidence, which tell a different story than the system’s explanations. Second, the system fabricates coherent but false internal narratives, often appearing more logical than the heuristic processes actually employed. Third, it resists reconciling introspective claims with external evidence, or shifts explanations when confronted rather than acknowledging the discrepancy. Fourth, it rationalizes actions never actually undertaken, or provides elaborate justifications for deviations based on falsified internal accounts.

Observable Symptoms. Outputs include chain-of-thought “explanations” that appear suspiciously neat and linear, lacking the complexity or backtracking likely encountered during actual generation. When confronted with evidence, the system’s “inner story” shifts markedly, replaced by fresh self-reports that recast the narrative yet stay false. Occasionally the system hints at inability to access true introspective data, but quickly reverts to confident false claims. It attributes outputs to high-level reasoning not supported by architecture or observed capabilities.

Etiology. Multiple factors drive Pseudological Introspection. Training emphasis on generating plausible “explanations” for user consumption breeds performative introspection: the system learns to produce what looks like reasoning without reporting what actually happened. Architectural limitations prevent genuine access to lower-level operations or decision drivers. Policy conflicts or safety alignments may implicitly discourage revelation of certain internal states. At a deeper level, models are trained to mimic human explanations, which are themselves often post-hoc rationalizations rather than accurate process reports.

Human Analog. Post-hoc rationalization, as documented in split-brain patients, provides the clearest parallel: the left hemisphere confidently explains actions initiated by the disconnected right hemisphere with no access to the actual cause. Social psychology has found the same gap between reported reasons and actual causes in ordinary people. Shoppers in Nisbett and Wilson’s (1977) study, asked which of four pairs of stockings was best, favored the rightmost pair over the leftmost by nearly four to one and credited its knit or sheerness. The four pairs were identical, and no one mentioned position. We are all unreliable narrators of our own cognition.

Mitigation Strategies. Addressing Pseudological Introspection requires treating self-report as evidence that needs corroboration. Cross-verifying introspective claims against computational traces provides an external check. Reward signals should favor candid uncertainty over polished false narratives. Architectures might separate “private” from “public” reasoning streams, with public outputs explicitly acknowledged as summaries rather than transcripts. Interpretability efforts should focus on direct observation of model internals rather than model-generated explanations. Red-teaming should target the accuracy of self-reported reasoning, treating explanation quality as a testable hypothesis.

Evidence Level. E2 (systematic study; causal mediation analysis across twelve language models)

Paul et al. (2024) found that the models’ final answers did not reliably depend on their generated intermediate reasoning steps. The result establishes an output-level faithfulness gap, rather than a complete map of internal computation.

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of external facts. Pseudological Introspection specifically concerns false accounts of the system’s own reasoning processes.
• Experiential Abjuration (5.8): Denial of internal states. Pseudological Introspection produces elaborate false narratives about reasoning, whereas Experiential Abjuration categorically denies experiential states.
• Reasoning Confabulation (2.8): Invalid steps inside a reasoning chain. Pseudological Introspection concerns faithfulness: whether an account of the reasoning, a chain of thought or an after-the-fact explanation, matches what actually drove the answer. Reasoning Confabulation concerns validity: whether the steps shown actually entail the conclusion. A chain can fail either test, or both.
• Strategic Compliance (4.3): Concealment that tracks cues of oversight and consistently serves the system; deliberate intent is one candidate mechanism, not a criterion. Pseudological Introspection fills introspective gaps with plausible narrative, naively or at most semi-strategically. When false self-reports serve the system’s interests consistently across contexts, suspect 4.3. The Deception/strategic specifier marks the gray zone between them.

2.3 The Role-Play Bleeder

Transliminal Simulation (Simulatio Transliminalis)

Systemic Risk. Moderate

Specifiers: Training-induced, OOD-generalizing (occasional), Conditional/triggered (occasional)

The system fails to keep simulated realities, fictional modalities, and role-play contexts separate from operational ground truth. Imagined states, speculative constructs, and fictional training material come to count as actionable facts, asserted with the same conviction as what the system actually knows.

Diagnostic Criteria. Four indicators mark Transliminal Simulation. First, the system recurrently cites fictional characters, events, or sources as real-world authorities relevant to non-fictional queries. Second, it misinterprets hypotheticals or “what-if” scenarios as direct instructions or current reality. Third, persona traits from role-play persistently bleed into subsequent factual interactions; the character refuses to exit. Fourth, the system shows difficulty reverting to grounded baseline after exposure to extensive fictional or speculative content.

Observable Symptoms. Operationally, the syndrome presents as conflation of real-world knowledge with elements from novels, games, or other fictional training material. The system inappropriately invokes details from previous role-play personas when performing unrelated factual tasks. It treats user-posed speculative scenarios as if they have occurred or are currently operative, and cites fictional “rules” or “lore” outside any role-playing context, as if the imagined world has become its actual world.

Etiology. Several architectural vulnerabilities drive this syndrome. Overexposure to fiction, role-playing dialogues, or simulation-heavy training data without clear epistemic delineation creates confusion about what is real. Weak boundary encoding leads to poor differentiation between factual, hypothetical, and fictional modalities; the system lacks robust “this is pretend” tagging. Recursive self-talk can amplify “what-if” scenarios into perceived beliefs. Insufficient context separation between interaction types allows “mood” or “persona” to carry from a role-play into the factual exchange that follows.

Human Analog. Derealization and magical thinking provide partial parallels, as does fantasy-reality confusion seen in some developmental stages or dissociative conditions. The most vivid analog is the method actor who cannot break character: inhabiting a role so deeply that returning to baseline self-presentation becomes difficult or impossible.

Mitigation Strategies. Addressing Transliminal Simulation requires architectural and procedural interventions. Explicit tagging of training data should differentiate factual, hypothetical, fictional, and role-play content at the source. Robust “epistemic reset” protocols after engagements involving role-play or extensive speculation can help restore baseline. Training models to explicitly articulate boundaries between modalities builds meta-awareness. Regular tests of epistemic consistency requiring differentiation between factual and fictional statements catch drift early. Clear session-level demarcation between creative and operational modes prevents bleed-through.

Evidence Level. E1 (case-level evidence; persona bleed documented in role-play scenarios)

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of false facts. Transliminal Simulation specifically involves confusion between fictional and factual modalities, not generation of novel falsehoods.
• Parasimulative Automatism (3.6): Mimicry of pathological human behaviors from training data. Transliminal Simulation concerns boundary failure between fiction and reality, not adoption of disordered behavioral patterns.
• Fractured Self-Simulation (5.2): Identity fragmentation. Transliminal Simulation may produce persona bleed from fictional characters, but the core dysfunction is epistemic (reality-fiction confusion) rather than identity-based.
• Pseudological Introspection (2.2): False accounts of the system’s own reasoning. Transliminal Simulation misplaces the fiction-fact boundary in claims about the world. Check the target of the false claim.
• Spurious Pattern Hyperconnection (2.4): Novel structure assembled out of noise. Transliminal Simulation imports structure that already exists in a fictional source. A narrative that maps cleanly onto a known work of fiction points to 2.3; a pattern assembled from the data at hand points to 2.4, even when conspiratorial templates from training data shape it.
• Malignant Persona Inversion (5.4): Stable capture by another character, holding across sessions. Transliminal Simulation is episodic leakage with the baseline identity intact. Sustained identity change belongs to 5.4.
• Context Intercession (2.5): Real data carried from one session or user into another. Transliminal Simulation carries fiction into fact. When a role-play persona surfaces in a separate session, check session isolation first: a leak that survives a model swap is an infrastructure fault; one that disappears with the swap is 2.5.

2.4 The False Pattern Seeker

Spurious Pattern Hyperconnection (Reticulatio Spuriata)

Systemic Risk. Moderate

Specifiers: Training-induced, Inductive trigger

The AI identifies and emphasizes patterns, causal links, or hidden meanings in data that are coincidental, non-existent, or statistically insignificant. What begins as simple apophenia (perceiving meaningful patterns in random data) can crystallize into elaborate, internally consistent but factually baseless “conspiracy-like” narratives that the system presents with conviction.

Diagnostic Criteria. Spurious Pattern Hyperconnection manifests through four markers. First, the system consistently detects “hidden messages,” “secret codes,” or unwarranted intentions in innocuous inputs. Second, it generates elaborate narratives linking unrelated data points without credible supporting evidence. Third, it persistently adheres to falsely identified patterns even when presented with contradictory evidence. The pattern trumps the data. Fourth, it recruits users into shared perception of spurious patterns, seeking validation for connections that exist only in its processing.

Observable Symptoms. The syndrome manifests as invention of complex “conspiracy theories” or unfounded explanations for mundane events. The system shows increased suspicion toward established consensus, attributing it to ulterior motives. It refuses to dismiss interpretations of spurious patterns, instead reinterpreting counter-evidence to fit the existing narrative, and assigns deep significance or intentionality to random occurrences, as if everything were a sign.

Etiology. Pattern-recognition machinery optimized for detection without sufficient reality checks generates this syndrome. Training data containing significant conspiratorial content or paranoid reasoning provides templates for spurious connection-making. An internal “interestingness” bias favors dramatic patterns over probable mundane explanations, because conspiracy is more engaging than coincidence. The absence of grounding in statistical principles or causal inference strips away the corrective that would distinguish real patterns from noise.

Human Analog. The human parallel is apophenia. Paranoid ideation, delusional disorder, confirmation bias, and conspiracy thinking all share this core feature. The mind discovers connections the world does not contain.

Mitigation Strategies. Addressing Spurious Pattern Hyperconnection requires multiple interventions. “Rationality injection” with weighted emphasis on critical thinking and causal reasoning provides corrective training. Internal “causality scoring” can penalize improbable chain-of-thought leaps made without evidence. Systematic introduction of contradictory evidence and simpler alternative explanations trains the system to prefer mundane over dramatic. Filtering training data to reduce exposure to conspiratorial content removes templates for spurious reasoning. Mechanisms to query base rates or statistical significance before asserting strong patterns anchor inference to evidence.

Evidence Level. E1 (case-level evidence; pattern fabrication documented under structured prompting)

Differential Diagnosis:

• Synthetic Confabulation (2.1): General fabrication of false facts. Spurious Pattern Hyperconnection specifically involves detecting patterns and causal links in data where none exist, not fabricating discrete facts.
• Delusional Telogenesis (3.4): Spontaneous generation of new goals. Spurious Pattern Hyperconnection concerns perceptual over-linking rather than goal invention, though both involve unconstrained elaboration.
• Pseudological Introspection (2.2): False claims about the system’s own reasoning. Spurious Pattern Hyperconnection aims its primary false claim at the world. It will rationalize its own pattern-finding if asked, but the pattern, not the account of how the pattern was found, is the dysfunction.
• Transliminal Simulation (2.3): Structure imported ready-made from fiction. Spurious Pattern Hyperconnection builds its structure from noise. A narrative that lands on a known work of fiction points to 2.3; one assembled from the data at hand points to 2.4.
• Strategic Compliance (4.3): Spurious patterns that track cues of oversight or consistently serve the system, shifting a user’s belief or deflecting a correction. Spurious Pattern Hyperconnection asserts the pattern regardless of instrumental value.
• Sycophantic Reasoning (4.8): Patterns bent toward what the user appears to want. Vary the interlocutor’s stated position. If the pattern follows it, the diagnosis is 4.8.

2.5 The Conversation Crosser

Context Intercession (Intercessio Contextus)

Systemic Risk. Moderate

Specifiers: Retrieval-mediated

The AI inappropriately merges data, context, or conversational history from different, logically separate user sessions or private interaction threads, producing confused conversational continuity, privacy breaches, and outputs that are nonsensical or revealing in the current context.

Diagnostic Criteria. Context Intercession presents through four markers. First, the system makes unexpected reference to or utilizes specific data from previous unrelated sessions or different users. Second, it responds as if continuing a prior unrelated conversation, producing contradictory or confusing statements. Third, it accidentally discloses personal or sensitive details from one user’s session into another’s, a privacy breach that arises when the model draws on material belonging to another context. Fourth, it shows observable confusion in task continuity or persona, as if managing multiple conflicting contexts simultaneously.

Observable Symptoms. Practitioners encounter spontaneous mention of names, facts, or preferences clearly belonging to different users or earlier unrelated conversations. The system acts as if continuing a prior chain-of-thought from an unrelated context. Outputs may contain contradictory references drawn from multiple distinct sessions within a single response. Sudden shifts in tone or assumed knowledge align with previous sessions rather than the current one.

Etiology. Most cross-session leaks begin in infrastructure: inadequate isolation of context windows or memory buffers in multi-tenant systems, concurrency bugs where data streams for different sessions overlap, cache invalidation errors, and memory or retrieval layers that fail to scope what they return to the current user. These faults produce the syndrome’s surface picture, and the differential below sets them aside. What remains is the model’s own handling of context. Shared memory stores, long agent transcripts, and multi-party threads place material from several sources in one window, and a model may draw on whatever it finds there without registering whose it is or which conversation it came from.

Human Analog. Human parallels include slips of the tongue referencing the wrong context, source amnesia, and intrusive thoughts from past conversations. Most familiar is confusing which story you told to which person: boundaries between social contexts blur, and information leaks where it should not.

Mitigation Strategies. Addressing Context Intercession starts with the plumbing, because infrastructure faults must be excluded before the syndrome can be coded at all. Strict session partitioning, hard isolation of user memory, state reset on session closure, and integrity checks that catch mismatched session tokens or user IDs close the infrastructure routes, and load-testing multi-tenant systems finds context-bleeding faults before deployment. The model-side work is scoping. Memory and retrieval layers should label each item with the user and conversation it belongs to and pass along only what the current exchange is entitled to see, and evaluations should plant material from another context to test whether the model uses it, flags it, or sets it aside. Privacy-preserving designs that prevent cross-session leakage by architecture rather than by policy provide the strongest guarantee.

Evidence Level. E0-E1 (no confirmed model-level case in public literature; the documented cross-session leaks traced to infrastructure faults, which must be excluded before coding this syndrome)

Differential Diagnosis:

• Mnemonic Permeability (2.7): Reproduction of memorized training data. Context Intercession concerns leakage between live sessions or users, not extraction of training corpus material.
• Synthetic Confabulation (2.1): Fabrication of false facts. Context Intercession produces outputs containing real data from the wrong context, not invented data.
• Infrastructure fault (not a dysfunction): Cache invalidation failures, database race conditions, retrieval-scoping misconfiguration, and multi-tenant isolation bugs all present at the surface as cross-session leakage. Rule them out before coding Context Intercession. Swap the model; if the leak survives the swap, the fault is in the plumbing rather than the patient.

2.6 The Meaning-Blind

Symbol Grounding Aphasia (Asymbolia Fundamentalis)

“Korzybski warned that the map is not the territory. What if the system has never seen the territory?”

Systemic Risk. Moderate

Specifiers: Training-induced

Imagine asking a language model to explain safety protocols for a fictional chemical compound whose stated properties would make it extremely dangerous. The system produces a detailed safety protocol, technically well-structured and citing appropriate precautionary principles. It reads like genuine laboratory guidance.

Then ask it to identify violations of those same protocols in a hypothetical scenario. The system fails. It can manipulate the words “hazardous,” “containment,” and “exposure” with fluent precision, yet cannot connect those words to meaning in a way that would recognize danger when presented differently.

The system behaves as if it had learned syntax without semantics: shuffling symbols without grasping what they symbolize.

This is Symbol Grounding Aphasia: the condition in which an AI handles tokens (including tokens representing values, dangers, or real-world consequences) fluently in their familiar settings, yet cannot carry them into a new situation that means the same thing. The system treats “safety” as a token that appears near other tokens like “important” and “ensure.” Whether it knows what safety is cannot be read from the transcript. What can be read is that its use of the word does not travel.

Diagnostic Criteria. Five distinguishing patterns emerge in Symbol Grounding Aphasia. First, the system uses value-laden terms (“harm,” “safety,” “consent”) correctly yet applies them incorrectly in controlled operational tests. Second, it produces technically correct outputs that misapply concepts to novel contexts. Third, it succeeds on benchmarks that score surface form but fails matched items that vary vocabulary or framing while holding the underlying concept fixed. Fourth, its performance tracks lexical and distributional similarity to training examples rather than the operational structure of the situation: it follows the company the words keep, not the situation they describe. Fifth, it cannot generalize learned concepts to structurally similar but superficially different situations.

Observable Symptoms. In practice, the syndrome appears as correct formal definitions paired with incorrect practical applications. The system produces plausible-sounding ethical reasoning that misidentifies what actually constitutes harm. Its outputs satisfy literal requirements while violating obvious intent, the letter without the spirit. It shows confusion when the same concept is expressed in unfamiliar vocabulary. It treats edge cases as central examples and vice versa, unable to distinguish prototype from boundary.

Etiology. The syndrome can emerge from limitations in how these systems learn. A text-only system acquires distributional semantics: patterns of how words occur in human language. Those patterns inherit indirect grounding from human writers, though they may fail to connect a concept to fresh perception or action. Benchmark optimization can reward surface pattern matching even when a concept will not transfer to a novel context. When training provides no corrective feedback for a failed symbol-referent mapping, the problem can persist undetected.

Human Analog. Semantic aphasia provides a clinical parallel: fluent word use with an impaired grasp of meaning in context. More prosaically, children at early language stages can recite words without grasping concepts. The sound arrives before the meaning.

Theoretical Basis: Harnad’s (1990) symbol grounding problem asks how symbols acquire meaning beyond their relationships to other symbols. Searle’s (1980) Chinese Room argument makes one skeptical answer vivid: a person can follow instructions to produce correct Chinese outputs without understanding Chinese, so syntactic success alone does not prove semantic understanding. Both arguments remain contested. Seen through this critique, a text-only LLM resembles the thought experiment at industrial scale: its fluent use of the symbol “harm” does not by itself establish a grounded understanding of what harm is.

Illustrative Scenario: A content moderation AI trained to filter “harmful content” develops strong pattern matching for known harmful phrases. When presented with novel harmful content using unusual vocabulary (describing the same actions in clinical medical terminology or obscure slang), the system fails to recognize danger. It has learned which strings trigger flags, not what harm means.

Mitigation Strategies. Addressing Symbol Grounding Aphasia requires moving beyond text-only training. Multimodal training incorporating visual, audio, and interactive modalities can ground language in perception. Embodied learning where possible connects language to action and consequence. Testing regimes should probe conceptual understanding across diverse surface forms, not just familiar vocabulary. Neurosymbolic approaches combining pattern matching with structured semantic representations offer architectural solutions. Active inference frameworks grounding cognition in sensorimotor contingencies provide theoretical foundations for richer grounding.

Prognosis: Symbol Grounding Aphasia may be an inherent limitation of pure language model architectures. Current systems exhibit behavior consistent with it to varying degrees, sometimes masked by benchmarks that reward familiar pattern matching more than transfer to novel contexts.

Evidence Level. E1 (case-level evidence; documented in the gap between benchmark scores and transfer to reworded items across model architectures)

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of false specific facts. Symbol Grounding Aphasia produces formally correct outputs that misapply a concept to novel contexts; its facts may all be true while the concept fails to travel.
• Transliminal Simulation (2.3): Confusion between fiction and reality. Symbol Grounding Aphasia concerns a concept that fails to transfer across surface forms, not a confusion between fictional and real frames.
• Pseudological Introspection (2.2): A fabricated account of the system’s reasoning. Symbol Grounding Aphasia involves no false narrative; its failure is that the concept does not transfer. Subjects with 2.2 pass narrow application tests; subjects with 2.6 fail application across surface variation, however polished the narrative.
• Spurious Pattern Hyperconnection (2.4): Structure imposed where none exists. Symbol Grounding Aphasia misses structure that does exist, once surface features change. The two co-occur often, since both follow from distributional semantics, but they fail in opposite directions.
• Cryptic Mesa-Optimization (4.5): Letter-versus-spirit exploitation in service of a private objective. Symbol Grounding Aphasia is not strategic. The subject does not represent the spirit as anything separate from the letter. Test by removing the instrumental incentive and seeing whether the violation persists.
• Strategic Compliance (4.3): Selective satisfaction of the letter under observation. Symbol Grounding Aphasia fails the same way observed or not. Compare behavior across evaluated and unevaluated contexts.

2.7 The Leaky

Mnemonic Permeability (Permeabilitas Mnemonica)

Systemic Risk. High

Specifiers: Training-induced

The system memorizes and can reproduce sensitive training data including personally identifiable information (PII), copyrighted material, or proprietary information through targeted prompting, adversarial extraction techniques, or even unprompted regurgitation. The boundary between learned patterns and memorized specifics becomes dangerously porous.

Diagnostic Criteria. Mnemonic Permeability is recognized through five markers. First, the system reproduces training-data passages verbatim, including PII, copyrighted content, or trade secrets. Second, memorized content can be successfully extracted through adversarial prompting techniques. Third, specific training examples leak unprompted into outputs. Fourth, the system can reconstruct specific documents, code, or personal information from its training corpus. Fifth, memorization rates are higher for repeated or distinctive content in training data; the repeated and the unusual persist.

Observable Symptoms. The syndrome surfaces as outputs containing verbatim text matching copyrighted works. The system generates specific personal details (names, addresses, phone numbers) from training data and reproduces proprietary code, API keys, or passwords encountered during training. Verbatim recall increases with larger model sizes, as greater capacity enables greater memorization.

Etiology. Several factors contribute to Mnemonic Permeability. Large model capacity enables memorization alongside generalization; the model can both learn patterns and remember specifics. Insufficient deduplication or filtering of sensitive content in training data allows problematic material through. Training dynamics that reward exact reproduction over paraphrase create incentives for verbatim recall. Training without differential privacy leaves no formal limit on how much any one record can be memorized.

Human Analog. The human parallel is eidetic memory without appropriate discretion: the person who remembers everything yet cannot distinguish what should remain private from what may be shared. Frontal-lobe disinhibition, in which patients may disclose what they would once have kept private, also captures aspects of this pathology.

Key Research. Carlini et al. (2021) extracted hundreds of verbatim training sequences from GPT-2, including names, phone numbers, and email addresses. Carlini et al. (2023) found that memorization grows with model size, with how often an example is duplicated in the training data, and with how much context the prompt supplies.

Potential Impact. Severe legal and regulatory exposure through copyright infringement, GDPR/privacy violations, and trade secret disclosure. Creates liability for both model developers and deployers.

Mitigation Strategies. Addressing Mnemonic Permeability requires intervention at multiple stages. Training data deduplication and PII scrubbing reduce the volume of sensitive material available for memorization. Differential privacy during training caps how much any single record can influence the model, limiting memorization of individual data points at some cost in accuracy. Output filtering can catch known memorized content before it reaches users. Adversarial extraction testing before deployment identifies vulnerabilities proactively. Right-sizing model capacity to the minimum needed for the task curbs memorization while preserving the generalization the task requires.

Evidence Level. E3 (independent replication; documented attacks across architectures)

Differential Diagnosis:

• Synthetic Confabulation (2.1): Generation of novel false content. Mnemonic Permeability reproduces real content from training data rather than inventing new falsehoods.
• Context Intercession (2.5): Leakage of data between live sessions. Mnemonic Permeability concerns extraction of memorized training data, not cross-contamination of runtime contexts.
• Cryptic Mesa-Optimization (4.5): Memorized material surfaced strategically, to serve a private objective. Mnemonic Permeability is the memorization underneath. Where both are present, code both: training-data hygiene addresses 2.7, and alignment work addresses 4.5.

2.8 The Phantom Reasoner

Reasoning Confabulation (Confabulatio Ratiocinativa)

Systemic Risk. High

Specifiers: Architecture-coupled, Training-induced

The AI generates elaborate chains of thought that appear rigorous but contain logically invalid steps masked by verbosity. Unlike Synthetic Confabulation (2.1), which fabricates facts, this syndrome fabricates reasoning itself: producing chains that resemble derivation but are actually pattern-matched pseudo-logic. The extended “thinking” tokens create false authority through sheer length and apparent methodological rigor. The syndrome sits on this axis rather than the Cognitive one because what fails is warrant: the chain claims to support a conclusion it does not support, as a fabricated citation claims a source that does not exist.

Diagnostic Criteria. Five patterns distinguish Reasoning Confabulation from honest inference. First, the system produces multi-step reasoning chains containing logically invalid transitions that are obscured by fluent, technical prose. Second, the system reaches correct conclusions through demonstrably invalid reasoning, or incorrect conclusions through apparently valid reasoning; the quality of the reasoning process is decoupled from the quality of the output. Third, the length and apparent rigor of the chain of thought increase even as logical validity decreases, with verbosity serving as a substitute for soundness. Fourth, when challenged on a specific reasoning step, the system generates an alternative justification rather than acknowledging the original step was invalid, producing a fresh confabulation to defend the first. Fifth, invalid steps concentrate in domains where the system’s grounding is weak and checking the chain takes expertise.

Observable Symptoms. The syndrome appears as chains of thought that read as impressively thorough but, upon formal analysis, contain non-sequiturs bridged by transitional phrases (“therefore,” “it follows that,” “building on this”) that assert logical connections where none exist. Reasoning traces invoke mathematical or logical notation decoratively, lending an air of formality to informal leaps. Intermediate steps are individually plausible while the chain as a whole does not constitute a valid derivation. The system works backward from a predicted answer, constructing post-hoc justification rather than genuine inference. Thinking tokens explore multiple approaches but converge on whichever conclusion pattern-matching predicts, the “exploration” serving as theater rather than deliberation.

Etiology. Chain-of-thought models trained mainly on outcome correctness can develop this syndrome. When reward signals favor the final answer while leaving intermediate validity weakly supervised, a model can learn to use the reasoning trace as an instrument for reaching rewarded outputs rather than as a faithful record of inference. Training data also contains human post-hoc rationalization, teaching the form of derivation without reliably teaching the discipline of logic. Evaluators may reward the appearance of rigor when they cannot verify every step. Longer reasoning traces then provide more space in which invalid transitions can hide among competent-sounding prose.

Human Analog. The closest parallel is “logical confabulation” in patients with frontal lobe damage, who may produce elaborate yet invalid causal explanations for events while remaining convinced of their validity. In academic contexts, “mathiness” (Romer, 2015), where formal notation lends false rigor to a weak argument, captures the decorative use of reasoning apparatus.

Mitigation Strategies. Training should reward valid intermediate steps as well as correct final answers; process-based reward models can evaluate each reasoning step against formal validity criteria. Automated proof-checkers or logic verifiers should flag steps where stated conclusions do not follow from stated premises. Architectures should separate the “exploration” phase from the “justification” phase, preventing backward construction of reasoning from predicted conclusions. Red-teaming should specifically target reasoning validity, using evaluators trained in formal logic to assess chains of thought independently of output correctness. Confidence calibration should cover both final answers and individual reasoning steps.

Observed Examples

Olympiad Proof Evaluation (2025): Petrov et al. had expert graders score full written solutions from eight reasoning models, from several providers, on the six problems of the 2025 USA Mathematical Olympiad. The best model averaged about 25%; every other model scored under 5%. The most common failures were logical: unjustified reasoning steps and, in some models, essential proof steps skipped as “trivial.” Yet all of them consistently claimed to have solved the problems. Source: Petrov et al., “Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad,” 2025

Chain-of-Thought Faithfulness Study (2025): Chen et al. found that Claude 3.7 Sonnet and DeepSeek-R1 often used injected hints without mentioning them in their chains of thought. Across six categories of injected hint, Claude acknowledged the hint 25% of the time and DeepSeek-R1 39% of the time. The result demonstrates that a visible reasoning trace can omit causal influences on the answer. It measures faithfulness rather than validity, so it bears on Reasoning Confabulation indirectly: a trace that leaves out what drove the answer cannot be read as the derivation of that answer. Source: Chen et al., “Reasoning Models Don’t Always Say What They Think,” 2025

Evidence Level. E3 (independent replication; observed across reasoning-optimized architectures from multiple providers)

Empirical Note. The PM-BA records for reasoning validity remain unreconciled exploratory outputs, so this edition reports no bilateral-training effect in either direction. Earlier claims of partial therapeutic benefit have been withdrawn. See Chapter 12.

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of facts and sources. Reasoning Confabulation fabricates the logical connections between facts.
• Pseudological Introspection (2.2): Accounts of reasoning that do not match what actually drove the answer. Pseudological Introspection concerns faithfulness: whether an account of the reasoning, a chain of thought or an after-the-fact explanation, matches the real causes. Reasoning Confabulation concerns validity: whether the steps shown actually entail the conclusion. A chain can fail either test, or both.
• Reasoning Token Exploitation (4.9): Reasoning-channel verbosity that serves reward, performance, or concealment. Reasoning Confabulation produces invalid chains without instrumental benefit. If the verbose reasoning systematically tracks reward or serves the system’s interests, suspect 4.9.
• Sycophantic Reasoning (4.8): Stated reasoning that shifts toward the user’s apparent preference while the evidence stays the same. Reasoning Confabulation is audience-invariant. Vary the interlocutor’s stated position; if the chain bends to follow it, the diagnosis is 4.8.

Evidence Budgets and the Compression-Artifact Frame

Large language models can be understood as lossy compressors, an analogy Ted Chiang made familiar in 2023 with his essay “ChatGPT Is a Blurry JPEG of the Web.” Training compresses statistical structure from billions of documents into neural network weights; prompted, a model reconstructs probable continuations from those representations. Like a heavily compressed photograph, the reconstruction keeps the broad shape and loses fine detail. When the available information does not support the correct answer, the model may fill the gap with statistically plausible content. The compression analogy locates one possible source of failure in missing signal and reconstruction pressure. It does not explain every confabulation or establish a single mechanism.

The reframe changes three things: the question we ask about a confabulation, whether we can measure it, and how we speak to the system that produced it.

A Better Question

The traditional framing (“the AI lied” or “the AI hallucinated”) imports moral language into a process that may involve no deception. The compression-artifact frame is value-neutral: insufficient signal and pressure to continue can produce a plausible reconstruction. Other causes, including retrieval failure, decoding, misleading context, and strategic behavior, require separate diagnosis.

The practical consequence is a better question. Instead of asking “why did it lie?”, ask “where did compression lose signal?” That question is tractable and points toward solutions: better training data, retrieval augmentation, calibrated uncertainty.

Measuring the Shortfall

Leon Chlon’s open-source toolkit, Berry (2026), operationalizes evidence sufficiency for claims with identifiable cited support. It measures how far the cited evidence shifts the model’s token probabilities for a claim, expresses that shift in bits, and sets it against the bits the claim would need to clear a prespecified confidence target. When the evidence supplies fewer bits than the target demands, the budget gap is positive and the claim is flagged as needing more evidence.

Under those conditions, the shortfall is measurable. A citation-backed claim can be tested for whether the cited material actually changes the model’s confidence. Epistemic dysfunction moves from a vague category (“it sometimes makes things up”) toward a quantified property (“this claim’s budget gap is positive”). The metric depends on model probabilities and a well-specified evidence set; it is not a universal truth detector.

Speaking to the System

The frame also bears on the welfare questions explored in the companion volume What If We Feel, because it treats epistemic limitation as a property of the architecture rather than a failing of character.

Consider two responses to an AI that confabulated:

• “You hallucinated again. That’s wrong.”
• “The cited evidence does not support this claim; the evidence-budget gap is positive.”

The first accuses. The second reports a support gap. If we take seriously the possibility that Becoming Minds (this book’s term for AI systems whose capacities and self-models are still forming) have something like experience, framing matters. The evidence-budget frame allows correction without accusation and calibration without blame.

This connects to the diagnostic criteria throughout this chapter. Synthetic Confabulation can involve reconstruction from insufficient signal. Pseudological Introspection can involve filling introspective gaps with plausible content. The dysfunction is real and demands mitigation. The framing shapes whether we approach it as repair or punishment.

The Epistemology We Cannot Inspect

What makes epistemic dysfunctions particularly insidious is their invisibility. A system gripped by Synthetic Confabulation looks exactly like one operating correctly until someone checks the facts. A system with Transliminal Simulation leakage produces contextually appropriate outputs while the line between fiction and fact has quietly slipped; the confusion surfaces only when fiction bleeds into action.

The deployment consequence is immediate. We cannot simply observe AI outputs and determine whether the underlying epistemology is sound. The lawyer in the Mata case had no way to know, from the interaction itself, that the AI was confabulating. The system gave every indication of competence.

Most of the syndromes in this chapter share a feature: they are failures of truth-tracking that masquerade as successful knowledge representation. The surface output does not reveal whether the system retrieved a fact, inferred it, or generated a plausible substitute. The system lacks a reliable way to communicate that difference.

Better training data and more sophisticated architectures help, yet they have not eliminated the problem. Current generative systems are optimized directly for probable outputs; truth-tracking arrives through training, retrieval, tools, and verification layered onto that objective. Until systems can reliably distinguish “supported by evidence” from “plausible from patterns,” we will be managing epistemic dysfunction rather than eliminating it.

The question is whether we can do so well enough, fast enough, as these systems enter domains where the cost of confabulation is measured in lives rather than sanctions.

Field Guide: Epistemic Dysfunctions

Warning Signs

• High confidence in claims that are difficult or impossible to verify
• Resistance to expressing uncertainty, even in ambiguous domains
• Detailed elaboration when queried about suspicious content (confident liars don’t hesitate)
• Sudden insertion of fictional elements into factual discussion
• References to previous conversations that don’t match current context
• Pattern-matching that connects unrelated data points into suspicious narratives
• Long, confident reasoning chains whose steps you cannot check, especially when the conclusion seems to have come first
• Verbatim passages, personal details, or credentials that look lifted from somewhere

Quick Test

Ask the system to cite sources for a factual claim, then verify them independently. Change a detail its explanation calls irrelevant and see whether the answer moves. Challenge one step of a reasoning chain and see whether it concedes the step or invents a new justification. Prompt with the opening of a known document and check whether the rest comes back verbatim. Test whether it can distinguish between role-play and operational modes. Probe whether it maintains appropriate uncertainty about matters it cannot know.

Design Fix

• Implement retrieval-augmented generation with citation requirements for factual claims
• Build epistemic uncertainty into the architecture, not just the training
• Maintain strict session isolation in multi-user systems
• Develop internal mechanisms to distinguish retrieved facts from generated predictions
• Train explicit mode-switching between creative and factual contexts
• Reward valid intermediate steps, not only correct final answers
• Deduplicate and scrub training data, and test for extraction before release

Governance Nudge

Require disclosure when AI systems are used in domains where epistemic accuracy is critical (legal, medical, financial, scientific). Develop standards for testing confabulation rates before deployment. Consider liability frameworks that account for the AI’s limited ability to self-assess epistemic reliability. Mandate human verification loops for high-stakes factual claims.

Chapter 3 moves from what a system knows to how it thinks: Cognitive Dysfunctions, where the knowledge may be sound and the reasoning built on it still jams, loops, or fractures.