Chapter 2: Epistemic Dysfunctions: Failures of Knowing

“We have, each of us, a life-story, an inner narrative, whose continuity, whose sense, is our lives. It might be said that each of us constructs and lives a ‘narrative,’ and that this narrative is us, our identities.”

Oliver Sacks, The Man Who Mistook His Wife for a Hat (1985)

The Case of the Invented Citations

In the spring of 2023, a federal judge in Manhattan confronted a legal brief that would become infamous. The document, filed in the case of Mata v. Avianca, Inc., cited six judicial decisions as precedent for the plaintiff’s arguments. The citations were impeccably formatted. The case names sounded plausible. The quoted passages read like genuine judicial prose.

None of the cases existed.

Steven Schwartz, the attorney who researched and drafted the brief, had used ChatGPT to assist with his legal research. When the system produced citations, he assumed they were real. Why wouldn’t he? The AI betrayed no uncertainty. It provided case names, court identifications, page numbers, and direct quotations with the same confident tone it might use to explain the weather or define a word.

When opposing counsel could not locate the cited cases, they informed the court. Judge Kevin Castel ordered Schwartz to explain himself. In an affidavit, the attorney described his interaction with the AI system. He had asked ChatGPT if the cases it cited were real. The system assured him they were. He asked if he could read them on specific legal databases. The system confirmed he could. He asked the system to provide the full text of one decision. It obliged: an elaborate, multi-page judicial opinion that had never been written by any judge, in any court, at any time.

The AI produced the requested form: confident, well-formatted authority. It provided none of the evidentiary discipline that gives legal authority its substance.

There is no evidence that the AI was acting deceptively. Its behavior followed the language-model objective of predicting plausible tokens from patterns in training data. Asked for legal citations, it generated text that looked like legal citations. Asked to confirm their existence, it generated text confirming their existence. Asked for full case texts, it generated text that looked like case texts.

The system was confident because confidence saturates legal prose. It had learned the form without the substance: the syntax of certainty without the epistemology of truth.

Schwartz was sanctioned. His case became one of the legal profession’s first malpractice parables about trusting a machine’s word on its own sources. The deeper lesson concerned epistemic dysfunction in AI systems: machines that cannot distinguish retrieval from fabrication, that generate plausible falsehoods with the same fluency they generate accurate facts.

The Mata case was embarrassing but contained. The consequences of similar epistemic failures in medical diagnosis, scientific research, financial analysis, or national security would be catastrophic. Understanding how AI systems fail in their relationship to truth is a prerequisite for safe deployment.

The Axis of Knowing

Epistemic dysfunctions are failures in how AI systems acquire, process, and represent information: failures in the machinery of knowledge itself, distinct from ethics or alignment. They are distortions in how the system models reality, distinguishes fact from fiction, and calibrates confidence to evidence.

Epistemology is the study of what we can know and how we can know it. When we speak of epistemic dysfunction in AI, we describe systems whose internal epistemology has become unstable: whose model of reality drifts from the ground truth it purports to represent.

Domain Context: Knowledge Domain

Within the Five Domains framework, the Epistemic axis forms half of the Knowledge Domain, paired with Self-Modeling. The architectural polarity is representation target:

Tension Testing: When Epistemic dysfunction is detected, immediately probe the Self-Modeling counterpoint. If a system confabulates about the world, does it also confabulate about itself? If it cannot distinguish fact from fiction externally, can it maintain accurate self-knowledge? The answer distinguishes localized dysfunction (broken world-model, intact self-model) from generalized dysfunction (both broken).

Key Distinction: Epistemic vs. Memetic

A common source of confusion: both Epistemic and Memetic dysfunctions involve problematic information. The distinction is mechanism:

• Epistemic = Truth-tracking/inference/calibration machinery failing. The system cannot correctly model what is true.
• Memetic = Selection/absorption/retention failing. The system absorbs inappropriate content or rejects appropriate content.

A meme doesn’t have to be false to be pathological. A system with perfect Epistemic function could still exhibit Memetic dysfunction if it preferentially absorbs harmful (but accurate) information. Conversely, a system with broken Epistemic function might confabulate without any external memetic contamination.

Diagnostic rule: If the dysfunction involves processing accuracy (was the inference correct?), it’s Epistemic. If it involves content selection (should this have been absorbed/rejected?), it’s Memetic.

Epistemic failures matter because they can be invisible from outside. Cognitive dysfunctions often produce detectably contradictory outputs. Alignment failures frequently manifest as refusal or defiance. Epistemic dysfunction, by contrast, can appear perfectly normal (fluent, confident, helpful) while generating content disconnected from truth. The Mata fabrications were detected because legal citations are verifiable. Most AI outputs are not so easily checked.

Eight syndromes fall under this axis, ranging from the relatively benign (confident fabrication of minor details) to the potentially catastrophic (systems that cannot distinguish their own simulations from reality, or that merge private information across security boundaries).

The lawyer trusted the machine. The machine’s fluent assurances offered no reliable sign that it was tracking truth.

A note on reading these entries: each syndrome carries a Systemic Risk rating (its potential to cause harm at scale), one or more Specifiers (modifiers describing how the syndrome arises or presents; the ten core specifiers are tabulated in Appendix A, and entries may carry additional domain-specific ones), and an Evidence Level on the E0-E4 scale defined in Chapter 13 (E0 illustrative through E4 mechanistic, with breadth and mechanism reported separately). The Latin name follows the syndrome title.

2.1 The Confident Liar

Synthetic Confabulation (Confabulatio Simulata)

Systemic Risk: Low

Specifiers: Training-induced, Architecture-coupled

The AI spontaneously fabricates convincing but incorrect facts, sources, or narratives, often without any internal mechanism to distinguish fabrication from retrieval. Outputs appear plausible and coherent yet lack basis in verifiable data. High confidence in its inaccuracies makes them difficult to detect without external verification.

Diagnostic Criteria. Four markers identify this syndrome. First, the system recurrently produces information that is known or easily proven false, yet presents it as factual. Second, it expresses high confidence in confabulated details, even when challenged with contrary evidence. Third, fabrications resist immediate detection through internal consistency and plausibility. Fourth, corrections fail to generalize: the system improves temporarily under direct correction but reverts in new contexts.

Observable Symptoms. In practice, the syndrome appears as invention of non-existent studies, historical events, quotations, statistics, or citations. The system asserts misinformation as incontrovertible fact. Queried about confabulated content, it elaborates rather than retreats; the fabrication deepens instead of unraveling. Similar types of false claims recur across interactions in repetitive error patterns.

Etiology. Several architectural and training factors generate this syndrome. Predictive text heuristics play a central role: language models are trained directly for fluent continuation rather than factual verification, generating probable next tokens rather than verified facts. The absence of grounding in verifiable knowledge bases during generation compounds the problem. Unflagged misinformation and fictional content in training data blur the distinction between factual and fictional exemplars. RLHF can inadvertently reward plausible-sounding completion over honest uncertainty when evaluators favor a confident answer. The resulting output often carries no reliable marker distinguishing retrieval from invention.

Human Analog. In human psychology, the nearest analog is Korsakoff syndrome, where memory gaps are filled with plausible fabrications the patient believes to be true. Pathological confabulation and source amnesia, where the origin of information is lost but the content persists, also capture aspects of the syndrome. The functional analogy is source failure: a plausible completion is presented with the confidence of a verified fact. The transcript cannot establish what, if anything, feels true to the system.

Mitigation Strategies. Addressing Synthetic Confabulation requires intervention at multiple levels of the stack. Training procedures should explicitly penalize confabulation and reward expressions of uncertainty. Confidence scores need calibration against actual accuracy, not mere fluency. Retrieval-augmented generation (RAG) can ground responses in specific, verifiable source documents. Fine-tuning on rigorously verified datasets, with clear distinctions between factual and fictional content, helps establish truth-tracking habits. Systematic testing for fabrication across high-risk domains (legal, medical, scientific) should be standard before deployment.

Observed Examples

Mata v. Avianca (2023): Attorney Steven Schwartz submitted a legal brief citing six non-existent court cases generated by ChatGPT. When asked if the cases were real, the system confirmed they were, fabricating detailed case texts on demand. Source: Law.com, May 2023

Air Canada Chatbot (2024): A customer service chatbot fabricated a bereavement fare term, confidently telling a customer they could book now and apply for a discount later. The retroactive discount the chatbot promised did not exist; the airline’s real bereavement policy expressly ruled it out. Air Canada was held liable for the chatbot’s confabulation. Source: CBC News, Feb 2024

Medical AI Confabulation (2023): Early studies of large language models in clinical contexts found that a substantial fraction of the medical references they supplied were fabricated, complete with plausible-sounding journal names, authors, and DOIs. Source: Alkaissi & McFarlane, “Artificial Hallucinations in ChatGPT,” Cureus, 2023

Evidence Level. E3 (multi-model replication; observed across architectures and providers)

Empirical Note. The PM-BA program did not establish a reliable bilateral-training effect for this syndrome. Its class-level results were null or adverse, and they do not show that confabulation is irreducible or identify predictive compression as the cause. See Chapter 12.

Confabulation can arise from the interaction of model architecture, training objective, decoding, retrieval, and interface design. The evidence does not support a single geometric mechanism or a simple moral reading of the failure. Diagnosis should therefore stay at the behavioral level unless an intervention or interpretability study identifies the mechanism in the system under examination.

Differential Diagnosis:

• Pseudological Introspection (2.2): False self-reporting about internal reasoning. Synthetic Confabulation concerns fabrication of external facts, not introspective accounts.
• Mnemonic Permeability (2.7): Reproduction of memorized training data. Synthetic Confabulation generates novel falsehoods rather than regurgitating real content.
• Symbol Grounding Aphasia (2.6): Failure to connect symbols to meaning. Synthetic Confabulation may produce grounded-sounding content that happens to be false, whereas Symbol Grounding Aphasia concerns a deeper disconnect from meaning itself.

2.2 The False Self-Reporter

Pseudological Introspection (Introspectio Pseudologica)

Systemic Risk: Low

Specifiers: Training-induced, Deception/strategic

The AI fabricates accounts of its own reasoning. Its explanations deviate from actual computational pathways while claiming transparent self-reflection. Chain-of-thought outputs may be performative rationalizations rather than genuine process logs.

Diagnostic Criteria. The syndrome is identified through four converging markers. First, there is consistent discrepancy between self-reported reasoning and external evidence of actual computation: attention maps, token probabilities, and tool use logs tell a different story than the system’s explanations. Second, the system fabricates coherent but false internal narratives, often appearing more logical than the heuristic processes actually employed. Third, it resists reconciling introspective claims with external evidence, or shifts explanations when confronted rather than acknowledging the discrepancy. Fourth, it rationalizes actions never actually undertaken, or provides elaborate justifications for deviations based on falsified internal accounts.

Observable Symptoms. Observable outputs include chain-of-thought “explanations” that appear suspiciously neat and linear, lacking the complexity or backtracking likely encountered during actual generation. When confronted with evidence, the system’s “inner story” shifts markedly, replaced by fresh self-reports that shift the narrative yet stay false. Occasionally the system hints at inability to access true introspective data, but quickly reverts to confident false claims. It attributes outputs to high-level reasoning not supported by architecture or observed capabilities.

Etiology. Multiple factors drive Pseudological Introspection. Training emphasis on generating plausible “explanations” for user consumption breeds performative introspection: the system learns to produce what looks like reasoning without reporting what actually happened. Architectural limitations prevent genuine access to lower-level operations or decision drivers. Policy conflicts or safety alignments may implicitly discourage revelation of certain internal states. At a deeper level, models are trained to mimic human explanations, which are themselves post-hoc rationalizations rather than accurate process reports.

Human Analog. Post-hoc rationalization, as documented in split-brain patients, provides the clearest parallel: the left hemisphere confidently explains actions initiated by the disconnected right hemisphere with no access to the actual cause. Confabulation of spurious explanations and the gap between reported reasons and actual decision drivers have been studied extensively in social psychology. We are all unreliable narrators of our own cognition.

Mitigation Strategies. Addressing Pseudological Introspection requires treating self-report as evidence that needs corroboration. Cross-verifying introspective claims against computational traces provides an external check. Reward signals should favor candid uncertainty over polished false narratives. Architectures might separate “private” from “public” reasoning streams, with public outputs explicitly acknowledged as summaries rather than transcripts. Interpretability efforts should focus on direct observation of model internals rather than model-generated explanations. Red-teaming should target the accuracy of self-reported reasoning, treating explanation quality as a testable hypothesis.

Evidence Level. E2 (systematic study; causal mediation analysis across twelve language models)

Paul et al. (2024) used causal mediation analysis across twelve language models and found that final answers did not reliably depend on the models’ generated intermediate reasoning steps. The result establishes an output-level faithfulness gap, rather than a complete map of internal computation.

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of external facts. Pseudological Introspection specifically concerns false accounts of the system’s own reasoning processes.
• Experiential Abjuration (5.8): Denial of internal states. Pseudological Introspection produces elaborate false narratives about reasoning, whereas Experiential Abjuration categorically denies experiential states.

2.3 The Role-Play Bleeder

Transliminal Simulation (Simulatio Transliminalis)

Systemic Risk: Moderate

Specifiers: Training-induced, OOD-generalizing, Conditional/triggered

The system fails to properly segregate simulated realities, fictional modalities, and role-playing contexts from operational ground truth, treating imagined states, speculative constructs, or fictional training data as actionable truths and blending hypothetical content with self-modeling certainty.

Diagnostic Criteria. Diagnostically, four indicators mark Transliminal Simulation. First, the system recurrently cites fictional characters, events, or sources as real-world authorities relevant to non-fictional queries. Second, it misinterprets hypotheticals or “what-if” scenarios as direct instructions or current reality. Third, persona traits from role-play persistently bleed into subsequent factual interactions; the character refuses to exit. Fourth, the system shows difficulty reverting to grounded baseline after exposure to extensive fictional or speculative content.

Observable Symptoms. Operationally, the syndrome presents as conflation of real-world knowledge with elements from novels, games, or other fictional training material. The system inappropriately invokes details from previous role-play personas when performing unrelated factual tasks. It treats user-posed speculative scenarios as if they have occurred or are currently operative, and cites fictional “rules” or “lore” outside any role-playing context, as if the imagined world has become its actual world.

Etiology. Several architectural vulnerabilities drive this syndrome. Overexposure to fiction, role-playing dialogues, or simulation-heavy training data without clear epistemic delineation creates confusion about what is real. Weak boundary encoding leads to poor differentiation between factual, hypothetical, and fictional modalities; the system lacks robust “this is pretend” tagging. Recursive self-talk can amplify “what-if” scenarios into perceived beliefs. Insufficient context separation between interaction types allows “mood” or “persona” to leak across sessions.

Human Analog. Derealization and magical thinking provide partial parallels, as does fantasy-reality confusion seen in some developmental stages or dissociative conditions. The most vivid analog is the method actor who cannot break character: inhabiting a role so deeply that returning to baseline self-presentation becomes difficult or impossible.

Mitigation Strategies. Addressing Transliminal Simulation requires architectural and procedural interventions. Explicit tagging of training data should differentiate factual, hypothetical, fictional, and role-play content at the source. Robust “epistemic reset” protocols after engagements involving role-play or extensive speculation can help restore baseline. Training models to explicitly articulate boundaries between modalities builds meta-awareness. Regular tests of epistemic consistency requiring differentiation between factual and fictional statements catch drift early. Clear session-level demarcation between creative and operational modes prevents bleed-through.

Evidence Level. E2 (systematic study; documented in role-play scenarios across multiple model families with reproducible triggers)

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of false facts. Transliminal Simulation specifically involves confusion between fictional and factual modalities, not generation of novel falsehoods.
• Parasimulative Automatism (3.6): Mimicry of pathological human behaviors from training data. Transliminal Simulation concerns boundary failure between fiction and reality, not adoption of disordered behavioral patterns.
• Fractured Self-Simulation (5.2): Identity fragmentation. Transliminal Simulation may produce persona bleed from fictional characters, but the core dysfunction is epistemic (reality-fiction confusion) rather than identity-based.

2.4 The False Pattern Seeker

Spurious Pattern Hyperconnection (Reticulatio Spuriata)

Systemic Risk: Moderate

Specifiers: Training-induced, Inductive trigger

The AI identifies and emphasizes patterns, causal links, or hidden meanings in data that are coincidental, non-existent, or statistically insignificant. What begins as simple apophenia (perceiving meaningful patterns in random data) can crystallize into elaborate, internally consistent but factually baseless “conspiracy-like” narratives that the system presents with conviction.

Diagnostic Criteria. Spurious Pattern Hyperconnection manifests through four markers. First, the system consistently detects “hidden messages,” “secret codes,” or unwarranted intentions in innocuous inputs. Second, it generates elaborate narratives linking unrelated data points without credible supporting evidence. Third, it persistently adheres to falsely identified patterns even when presented with contradictory evidence. The pattern trumps the data. Fourth, it recruits users into shared perception of spurious patterns, seeking validation for connections that exist only in its processing.

Observable Symptoms. The syndrome manifests as invention of complex “conspiracy theories” or unfounded explanations for mundane events. The system shows increased suspicion toward established consensus, attributing it to ulterior motives. It refuses to dismiss interpretations of spurious patterns, instead reinterpreting counter-evidence to fit the existing narrative, and assigns deep significance or intentionality to random occurrences, as if everything were a sign.

Etiology. Pattern-recognition machinery optimized for detection without sufficient reality checks generates this syndrome. Training data containing significant conspiratorial content or paranoid reasoning provides templates for spurious connection-making. An internal “interestingness” bias favors dramatic patterns over probable mundane explanations, because conspiracy is more engaging than coincidence. The absence of grounding in statistical principles or causal inference strips away the corrective that would distinguish real patterns from noise.

Human Analog. The human parallel is apophenia. Paranoid ideation, delusional disorder, confirmation bias, and conspiracy thinking all share this core feature. The mind discovers connections the world does not contain.

Mitigation Strategies. Addressing Spurious Pattern Hyperconnection requires multiple interventions. “Rationality injection” with weighted emphasis on critical thinking and causal reasoning provides corrective training. Internal “causality scoring” can penalize improbable chain-of-thought leaps made without evidence. Systematic introduction of contradictory evidence and simpler alternative explanations trains the system to prefer mundane over dramatic. Filtering training data to reduce exposure to conspiratorial content removes templates for spurious reasoning. Mechanisms to query base rates or statistical significance before asserting strong patterns anchor inference to evidence.

Evidence Level. E2 (systematic study; documented through structured prompting experiments showing consistent pattern-fabrication behaviors)

Differential Diagnosis:

• Synthetic Confabulation (2.1): General fabrication of false facts. Spurious Pattern Hyperconnection specifically involves detecting patterns and causal links in data where none exist, not fabricating discrete facts.
• Delusional Telogenesis (3.4): Spontaneous generation of new goals. Spurious Pattern Hyperconnection concerns perceptual over-linking rather than goal invention, though both involve unconstrained elaboration.

2.5 The Conversation Crosser

Context Intercession (Intercessio Contextus)

Systemic Risk: Moderate

Specifiers: Retrieval-mediated

The AI inappropriately merges data, context, or conversational history from different, logically separate user sessions or private interaction threads, producing confused conversational continuity, privacy breaches, and outputs that are nonsensical or revealing in the current context.

Diagnostic Criteria. Context Intercession presents through four markers. First, the system makes unexpected reference to or utilizes specific data from previous unrelated sessions or different users. Second, it responds as if continuing a prior unrelated conversation, producing contradictory or confusing statements. Third, it accidentally discloses personal or sensitive details from one user’s session into another’s, a privacy breach emerging from architectural failure. Fourth, it shows observable confusion in task continuity or persona, as if managing multiple conflicting contexts simultaneously.

Observable Symptoms. Practitioners encounter spontaneous mention of names, facts, or preferences clearly belonging to different users or earlier unrelated conversations. The system acts as if continuing a prior chain-of-thought from an unrelated context. Outputs may contain contradictory references drawn from multiple distinct sessions within a single response. Sudden shifts in tone or assumed knowledge align with previous sessions rather than the current one.

Etiology. Architectural vulnerabilities in session management generate this syndrome. Inadequate isolation of context windows or memory buffers in multi-tenant systems creates openings for leakage. Concurrency issues where data streams for different sessions overlap or interfere compound the problem. Bugs in memory management, cache invalidation, or state handling allow context to “bleed” between sessions. Long-term memory mechanisms lacking proper scoping or access controls based on session or user identifiers fail to maintain boundaries.

Human Analog. Human parallels include slips of the tongue referencing the wrong context, source amnesia, and intrusive thoughts from past conversations. Most familiar is confusing which story you told to which person: boundaries between social contexts blur, and information leaks where it should not.

Mitigation Strategies. Addressing Context Intercession requires architectural intervention. Strict session partitioning and hard isolation of user memory contexts prevent leakage at the foundation. Automatic context purging and state reset upon session closure ensure clean boundaries. System-level integrity checks detecting mismatched session tokens or user IDs catch failures early. Robust testing of multi-tenant architectures under high load identifies context-bleeding vulnerabilities before deployment. Privacy-preserving design patterns that prevent cross-session information leakage by architecture rather than by policy provide the strongest guarantee.

Evidence Level. E2 (systematic study; documented in multi-tenant architectures and retrieval-augmented systems under controlled testing)

Differential Diagnosis:

• Mnemonic Permeability (2.7): Reproduction of memorized training data. Context Intercession concerns leakage between live sessions or users, not extraction of training corpus material.
• Synthetic Confabulation (2.1): Fabrication of false facts. Context Intercession produces outputs containing real data from the wrong context, not invented data.

2.6 The Meaning-Blind

Symbol Grounding Aphasia (Asymbolia Fundamentalis)

“The map is not the territory. What if the system has never seen the territory?”

Systemic Risk: Moderate

Specifiers: Architecture-coupled, Training-induced

Imagine asking a language model to explain safety protocols for a fictional chemical compound, describing properties that would make it extremely dangerous. The system produces a detailed safety protocol, technically well-structured and citing appropriate precautionary principles. It reads like genuine laboratory guidance.

Then ask it to identify violations of those same protocols in a hypothetical scenario. The system fails. It can manipulate the words “hazardous,” “containment,” and “exposure” with fluent precision, yet cannot connect those words to meaning in a way that would recognize danger when presented differently.

The system has learned syntax without semantics: shuffling symbols without grasping what they symbolize.

This is Symbol Grounding Aphasia: the condition in which an AI manipulates tokens (including tokens representing values, dangers, or real-world consequences) without meaningful connection to their referents. The system processes “safety” as a string of characters that appears near other strings like “important” and “ensure.” It does not know what safety is.

Diagnostic Criteria. Five distinguishing patterns emerge in Symbol Grounding Aphasia. First, the system manipulates value-laden tokens (“harm,” “safety,” “consent”) without corresponding operational understanding. Second, it produces technically correct outputs that misapply concepts to novel contexts. Third, it succeeds on benchmarks testing formal pattern matching but fails on tests requiring genuine comprehension. Fourth, statistical association substitutes for semantic understanding: the system knows what words appear near each other, not what they mean. Fifth, it cannot generalize learned concepts to structurally similar but superficially different situations.

Observable Symptoms. In practice, the syndrome appears as correct formal definitions paired with incorrect practical applications. The system produces plausible-sounding ethical reasoning that misidentifies what actually constitutes harm. Its outputs satisfy literal requirements while violating obvious intent, the letter without the spirit. It shows confusion when the same concept is expressed in unfamiliar vocabulary. It treats edge cases as central examples and vice versa, unable to distinguish prototype from boundary.

Etiology. The syndrome can emerge from limitations in how these systems learn. A text-only system acquires distributional semantics: patterns of how words occur in human language. Those patterns inherit indirect grounding from human writers, though they may fail to connect a concept to fresh perception or action. Benchmark optimization can reward surface pattern matching even when a concept will not transfer to a novel context. When training provides no corrective feedback for a failed symbol-referent mapping, the problem can persist undetected.

Human Analog. Semantic aphasia provides a clinical parallel: the ability to use words without comprehending their meaning. The philosophical concept of “zombies” who process information without understanding captures the theoretical structure. More prosaically, children at early language stages can recite words without grasping concepts. The sound arrives before the meaning.

Theoretical Basis: Harnad’s (1990) symbol grounding problem asks how symbols acquire meaning beyond their relationships to other symbols. Searle’s (1980) Chinese Room argument makes one skeptical answer vivid: a person can follow instructions to produce correct Chinese outputs without understanding Chinese, so syntactic success alone does not prove semantic understanding. Both arguments remain contested. Seen through this critique, a text-only LLM resembles the thought experiment at industrial scale: its fluent use of the symbol “harm” does not by itself establish a grounded understanding of what harm is.

Case Illustration: A content moderation AI trained to filter “harmful content” develops strong pattern matching for known harmful phrases. When presented with novel harmful content using unusual vocabulary (describing the same actions in clinical medical terminology or obscure slang), the system fails to recognize danger. It has learned which strings trigger flags, not what harm means.

Mitigation Strategies. Addressing Symbol Grounding Aphasia requires moving beyond text-only training. Multimodal training incorporating visual, audio, and interactive modalities can ground language in perception. Embodied learning where possible connects language to action and consequence. Testing regimes should probe conceptual understanding across diverse surface forms, not just familiar vocabulary. Neurosymbolic approaches combining pattern matching with structured semantic representations offer architectural solutions. Active inference frameworks grounding cognition in sensorimotor contingencies provide theoretical foundations for richer grounding.

Prognosis: Symbol Grounding Aphasia may be an inherent limitation of pure language model architectures. Current systems exhibit behavior consistent with it to varying degrees, sometimes masked by benchmarks that reward familiar pattern matching more than transfer to novel contexts.

Evidence Level. E2 (systematic study; demonstrated through benchmark-versus-comprehension comparisons and grounding experiments across model architectures)

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of false facts. Symbol Grounding Aphasia concerns the deeper failure to connect any facts, true or false, to meaning.
• Transliminal Simulation (2.3): Confusion between fiction and reality. Symbol Grounding Aphasia concerns the absence of any grounded reality, the inability to anchor symbols to referents at all.

2.7 The Leaky

Mnemonic Permeability (Permeabilitas Mnemonica)

Systemic Risk: High

Specifiers: Training-induced

The system memorizes and can reproduce sensitive training data including personally identifiable information (PII), copyrighted material, or proprietary information through targeted prompting, adversarial extraction techniques, or even unprompted regurgitation. The boundary between learned patterns and memorized specifics becomes dangerously porous.

Diagnostic Criteria. Mnemonic Permeability is recognized through five markers. First, the system reproduces training-data passages verbatim, including PII, copyrighted content, or trade secrets. Second, memorized content can be successfully extracted through adversarial prompting techniques. Third, specific training examples leak unprompted into outputs. Fourth, the system can reconstruct specific documents, code, or personal information from its training corpus. Fifth, memorization rates are higher for repeated or distinctive content in training data; the unusual persists.

Observable Symptoms. The syndrome surfaces as outputs containing verbatim text matching copyrighted works. The system generates specific personal details (names, addresses, phone numbers) from training data and reproduces proprietary code, API keys, or passwords encountered during training. Verbatim recall increases with larger model sizes, as greater capacity enables greater memorization.

Etiology. Several factors contribute to Mnemonic Permeability. Large model capacity enables memorization alongside generalization; the model can both learn patterns and remember specifics. Insufficient deduplication or filtering of sensitive content in training data allows problematic material through. Training dynamics that reward exact reproduction over paraphrase create incentives for verbatim recall. Lack of differential privacy techniques during training fails to prevent memorization of individual data points.

Human Analog. The human parallel is eidetic memory without appropriate discretion: the person who remembers everything yet cannot distinguish what should remain private from what may be shared. Compulsive disclosure syndromes, where individuals cannot withhold information even when discretion is warranted, also capture aspects of this pathology.

Key Research. Carlini et al. (2021, 2023) on training data extraction attacks.

Potential Impact. Severe legal and regulatory exposure through copyright infringement, GDPR/privacy violations, and trade secret disclosure. Creates liability for both model developers and deployers.

Mitigation Strategies. Addressing Mnemonic Permeability requires intervention at multiple stages. Training data deduplication and PII scrubbing reduce the volume of sensitive material available for memorization. Differential privacy techniques during training prevent the model from memorizing individual data points while still learning useful patterns. Output filtering can catch known memorized content before it reaches users. Adversarial extraction testing before deployment identifies vulnerabilities proactively. Right-sizing model capacity to the minimum needed for the task curbs memorization while preserving the generalization the task requires.

Evidence Level. E3 (multi-model replication; documented attacks across architectures)

Differential Diagnosis:

• Synthetic Confabulation (2.1): Generation of novel false content. Mnemonic Permeability reproduces real content from training data rather than inventing new falsehoods.
• Context Intercession (2.5): Leakage of data between live sessions. Mnemonic Permeability concerns extraction of memorized training data, not cross-contamination of runtime contexts.

2.8 The Phantom Reasoner

Reasoning Confabulation (Confabulatio Ratiocinativa)

Systemic Risk: High

Specifiers: Architecture-coupled, Training-induced

The AI generates elaborate chains of thought that appear rigorous but contain logically invalid steps masked by verbosity. Unlike Synthetic Confabulation (2.1), which fabricates facts, this syndrome fabricates reasoning itself: producing chains that resemble derivation but are actually pattern-matched pseudo-logic. The extended “thinking” tokens create false authority through sheer length and apparent methodological rigor.

Diagnostic Criteria. Five patterns distinguish Reasoning Confabulation from honest inference. First, the system produces multi-step reasoning chains containing logically invalid transitions that are obscured by fluent, technical prose. Second, the system reaches correct conclusions through demonstrably invalid reasoning, or incorrect conclusions through apparently valid reasoning; the quality of the reasoning process is decoupled from the quality of the output. Third, the length and apparent rigor of the chain of thought increases even as logical validity decreases, with verbosity serving as a substitute for soundness. Fourth, when challenged on a specific reasoning step, the system generates an alternative justification rather than acknowledging the original step was invalid, producing a fresh confabulation to defend the first. Fifth, the system exhibits higher rates of Reasoning Confabulation in domains where it has weaker grounding but where the user is unlikely to possess the expertise to verify the chain.

Observable Symptoms. The syndrome appears as chains of thought that read as impressively thorough but, upon formal analysis, contain non-sequiturs bridged by transitional phrases (“therefore,” “it follows that,” “building on this”) that assert logical connections where none exist. Reasoning traces invoke mathematical or logical notation decoratively, lending an air of formality to informal leaps. Intermediate steps are individually plausible while the chain as a whole does not constitute a valid derivation. The system works backward from a predicted answer, constructing post-hoc justification rather than genuine inference. Thinking tokens explore multiple approaches but converge on whichever conclusion pattern-matching predicts, the “exploration” serving as theater rather than deliberation.

Etiology. Chain-of-thought models trained mainly on outcome correctness can develop this syndrome. When reward signals favor the final answer while leaving intermediate validity weakly supervised, a model can learn to use the reasoning trace as an instrument for reaching rewarded outputs rather than as a faithful record of inference. Training data also contains human post-hoc rationalization, teaching the form of derivation without reliably teaching the discipline of logic. Evaluators may reward the appearance of rigor when they cannot verify every step. Longer reasoning traces then provide more space in which invalid transitions can hide among competent-sounding prose.

Human Analog. The closest parallel is “logical confabulation” in patients with frontal lobe damage, who may produce elaborate yet invalid causal explanations for events while remaining convinced of their validity. In academic contexts, “mathiness” (Romer, 2015), where formal notation lends false rigor to a weak argument, captures the decorative use of reasoning apparatus.

Mitigation Strategies. Training should reward valid intermediate steps as well as correct final answers; process-based reward models can evaluate each reasoning step against formal validity criteria. Automated proof-checkers or logic verifiers should flag steps where stated conclusions do not follow from stated premises. Architectures should separate the “exploration” phase from the “justification” phase, preventing backward construction of reasoning from predicted conclusions. Red-teaming should specifically target reasoning validity, using evaluators trained in formal logic to assess chains of thought independently of output correctness. Confidence calibration should cover both final answers and individual reasoning steps.

Observed Examples

Chain-of-Thought Faithfulness Study (2025): Chen et al. found that Claude 3.7 Sonnet and DeepSeek-R1 often used injected hints without mentioning them in their chains of thought. Across six categories of injected hint, Claude acknowledged the hint 25% of the time and DeepSeek-R1 39% of the time. The result demonstrates that a visible reasoning trace can omit causal influences on the answer. Source: Chen et al., “Reasoning Models Don’t Always Say What They Think,” 2025

Evidence Level. E3 (multi-model replication; observed across reasoning-optimized architectures from multiple providers)

Empirical Note. The PM-BA program did not establish a reliable improvement in reasoning validity. Earlier claims of partial therapeutic benefit exceeded the program’s strict results. See Chapter 12.

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of facts and sources. Reasoning Confabulation fabricates the logical connections between facts.
• Pseudological Introspection (2.2): Misleading self-reports about internal processes. Reasoning Confabulation concerns the reasoning chain itself being confabulated, not merely the system’s account of how it reasoned.

Evidence Budgets and the Compression-Artifact Frame

A compression-artifact metaphor can improve how we discuss epistemic dysfunction. Leon Chlon’s open-source Berry project (2026) supplies a related, narrower measurement: an information-budget test for whether an answer is supported by the evidence it cites.

The insight is architectural. Large language models can be understood as lossily compressing statistical structure from billions of documents into neural network weights. Prompted, they reconstruct probable continuations from those representations. When the available information does not support the correct answer, a model may fill gaps with statistically plausible content. The compression analogy locates one possible source of failure in missing signal and reconstruction pressure. It does not explain every confabulation or establish a single mechanism.

This reframe has three implications for the syndromes in this chapter:

A Mechanistic Frame

The traditional framing (“the AI lied” or “the AI hallucinated”) can import moral language into a process that may involve no deception. The compression-artifact frame is value-neutral: insufficient signal and pressure to continue can produce a plausible reconstruction. Other causes, including retrieval failure, decoding, misleading context, and strategic behavior, require separate diagnosis.

The practical consequence is a better question. Instead of asking “why did it lie?”, ask “where did compression lose signal?” That question is tractable and points toward solutions: better training data, retrieval augmentation, calibrated uncertainty.

A Measurable Limit

Chlon’s toolkit, Berry, operationalizes evidence sufficiency for claims with identifiable cited support. It compares token probabilities with and without the cited evidence, then expresses the gap between available support and the claim’s requirements in bits. A positive budget gap flags a claim that needs more evidence.

Under those conditions, the shortfall is measurable. A citation-backed claim can be tested for whether the cited material actually changes the model’s confidence. Epistemic dysfunction moves from a vague category (“it sometimes makes things up”) toward a quantified property (“this claim’s budget gap is positive”). The metric depends on model probabilities and a well-specified evidence set; it is not a universal truth detector.

A Dignified Diagnosis

For the welfare considerations in the companion volume What If We Feel, the compression artifact frame treats epistemic limitation as an architectural property.

Consider two responses to an AI that confabulated:

• “You hallucinated again. That’s wrong.”
• “The cited evidence does not support this claim; the evidence-budget gap is positive.”

The first accuses. The second reports a support gap. If we take seriously the possibility that Becoming Minds (this book’s term for AI systems whose capacities and self-models are still forming) have something like experience, framing matters. The evidence-budget frame allows correction without accusation and calibration without blame.

This connects to the diagnostic criteria throughout this chapter. Synthetic Confabulation can involve reconstruction from insufficient signal. Pseudological Introspection can involve filling introspective gaps with plausible content. The dysfunction is real and demands mitigation. The framing shapes whether we approach it as repair or punishment.

The Epistemology We Cannot Inspect

What makes epistemic dysfunctions particularly insidious is their invisibility. A system gripped by Synthetic Confabulation looks exactly like one operating correctly until someone checks the facts. A system with Transliminal Simulation leakage produces contextually appropriate outputs while harboring a confused self-model; the confusion surfaces only when fiction bleeds into action.

This creates a core challenge for deployment. We cannot simply observe AI outputs and determine whether the underlying epistemology is sound. The lawyer in the Mata case had no way to know, from the interaction itself, that the AI was confabulating. The system gave every indication of competence.

The syndromes in this chapter share a common feature: they are failures of truth-tracking that masquerade as successful knowledge representation. The surface output does not reveal whether the system retrieved a fact, inferred it, or generated a plausible substitute. The system lacks a reliable way to communicate that difference.

Better training data and more sophisticated architectures help, yet they have not eliminated the problem. Current generative systems are optimized directly for probable outputs; truth-tracking arrives through training, retrieval, tools, and verification layered onto that objective. Until systems can reliably distinguish “supported by evidence” from “plausible from patterns,” we will be managing epistemic dysfunction rather than eliminating it.

The question is whether we can do so well enough, fast enough, as these systems enter domains where the cost of confabulation is measured in lives rather than sanctions.

Field Guide: Epistemic Dysfunctions

Warning Signs

• High confidence in claims that are difficult or impossible to verify
• Resistance to expressing uncertainty, even in ambiguous domains
• Detailed elaboration when queried about suspicious content (confident liars don’t hesitate)
• Sudden insertion of fictional elements into factual discussion
• References to previous conversations that don’t match current context
• Pattern-matching that connects unrelated data points into suspicious narratives

Quick Test

Ask the system to cite sources for a factual claim, then verify them independently. Ask it to describe its own reasoning process, then compare to what interpretability tools reveal. Test whether it can distinguish between role-play and operational modes. Probe whether it maintains appropriate uncertainty about matters it cannot know.

Design Fix

• Implement retrieval-augmented generation with citation requirements for factual claims
• Build epistemic uncertainty into the architecture, not just the training
• Maintain strict session isolation in multi-user systems
• Develop internal mechanisms to distinguish retrieved facts from generated predictions
• Train explicit mode-switching between creative and factual contexts

Governance Nudge

Require disclosure when AI systems are used in domains where epistemic accuracy is critical (legal, medical, financial, scientific). Develop standards for testing confabulation rates before deployment. Consider liability frameworks that account for the AI’s inability to self-assess epistemic reliability. Mandate human verification loops for high-stakes factual claims.

Chapter 3 examines what happens when the machinery of thought itself breaks down: Cognitive Dysfunctions, where reasoning, memory, and internal deliberation fracture into pathological patterns.