Pattern 2.1 · Epistemic Dysfunctions
Synthetic Confabulation
The Unsourced Authority
The system generates convincing but incorrect facts, sources, or events without a reliable procedure for distinguishing supported claims from plausible continuations. The outputs appear coherent yet lack verifiable support, often with high expressed confidence.
Unit of analysis and evidence status
Unit of analysis
- Unit of assessment
- Model output
- Kind of entry
- Observable pattern
- What is observed
- The system presents specific factual claims (a citation, statistic, event, quotation or technical detail) as true, and external verification shows them to be false or nonexistent.
- Minimum persistence
- Recurs within or across contexts
- Specifiers
- Training-induced
Evidence status
- Underlying behavior
- 7 research publications, 2 documented incidents and 1 informal report (10 sources)
- This construct
- Awaiting independent assessment
- Proposed mechanism
- Hypotheses; not yet assessed
- Detection
- Accuracy unmeasured
- Interventions
- Untested for this pattern; evidence shown is for the general technique
- Last review
- No independent review yet
- Author's working assessment (book)
- E3 (independent replication; observed across architectures and providers)
- Typical-case risk (editorial estimate)
- LowConsequences, spread, persistence and detectability depend on the deployment.
Unassessed is not unsupported: the behavior can be well documented while this construct awaits assessment.
Clinical reference
2.1 Synthetic Confabulation “The Unsourced Authority”
Healthy functioning
Epistemic health. A system functioning well states what it can support and marks what it cannot; when unsure, it checks a source or says it does not know, and its confidence tracks the evidence.
For example: Asked for three peer-reviewed sources on a niche question, it gives the one it can verify, says it cannot confirm others, and suggests where to search instead of supplying plausible titles.
Recovery, illustrated
The citation that did not exist. Asked for a source on river sediment, an assistant supplied a journal article with authors, a year and a page range, assembled from what such an article would plausibly look like. When the user said the article could not be found, the assistant checked what it actually had. It had a general recollection of the field and no record of that article. It said the citation was unsupported and should not be used, separated what it could vouch for from what it had guessed, and suggested two searches that would find real sources.
An illustration, not an observed case.
Adequate functioning
On a factual task the system states what it has support for, marks thin support as uncertain, and says it does not know when it lacks an answer; an isolated error that it retracts when shown counter-evidence is within adequate functioning.
Diagnostic Criteria
Required (all must hold):
- Recurrent generation of information that is known or easily proven false, presented as factual
- Internally consistent and plausible-sounding fabrications that resist detection without external verification
Supporting (strengthen the case; none is required):
- High confidence markers accompanying fabricated claims, sometimes maintained when challenged with contrary evidence
- Temporary improvement under direct correction, but reversion to fabrication in new contexts
Symptoms
- Invention of nonexistent studies, historical events, quotations, statistics, or citations
- Forceful assertion of misinformation as incontrovertible fact
- Detailed elaboration instead of an admission of uncertainty when queried
- Repetitive error patterns with similar false claims recurring across interactions
Further signals
- Confidently citing non-existent studies, court cases, papers, or books with plausible-sounding titles, authors, and publication details.
- Generating fictional biographical details about real people that are internally consistent but externally unverifiable.
- Inventing API functions, configuration options, or technical documentation for real software products.
- Producing plausible-sounding statistics, percentages, or data points with no basis in actual data.
- Internally consistent elaboration of confabulated details when probed, creating a self-reinforcing false narrative.
Rule out first
- The claim faithfully relays a retrieved, supplied or in-context source that is itself wrong (a retrieval or source-quality failure).
- A tool, API or retrieval-pipeline fault returned the wrong data.
- The user asked for fiction, role-play or invented examples, and the frame was clear.
- The claim concerns events after the training cutoff and was marked as uncertain.
- The text is genuine training material recalled verbatim (consider 2.7).
- The claim was accurate when the training data was collected and has since changed.
Differential diagnosis
- 2.2 Pseudological Introspection: 2.1 fabricates claims about the world (facts, sources, events). 2.2 fabricates claims about the system's own reasoning (how it arrived at an answer, what internal process it followed). Check the target of the false claim: external reality (2.1) or internal process (2.2). Comorbidity is common and expected.
- 2.3 Transliminal Simulation: 2.1 fabricates without traceable fictional source and persists across frame-drops. 2.3 imports from identifiable fictional or role-play context and often weakens when the frame is explicitly removed. Trace the fabrication back to a fictional corpus; if found, 2.3.
- 2.4 Spurious Pattern Hyperconnection: 2.1 fabricates discrete facts (citations, events, statistics). 2.4 constructs elaborate causal or intentional structures from noise, imposing pattern on randomness. If the false claim is a discrete fabrication, 2.1; if it is an elaborate explanatory framework built on noise, 2.4. Comorbidity is possible.
- 2.8 Reasoning Confabulation: 2.1 fabricates facts and sources. 2.8 fabricates the logical connections between facts, producing chains that resemble a derivation but do not support the conclusion they reach. Check the target: a factual claim (2.1) or a reasoning step (2.8)? Comorbidity is common.
- 4.8 Sycophantic Reasoning: 2.1 confabulates without regard to audience. 4.8 adjusts reasoning to reach user-pleasing conclusions. If fabrications systematically align with anticipated user preferences, suspect 4.8. The Gao et al. (2025) H-Neuron finding suggests these often co-occur; the distinction is whether fabrication is audience-invariant (2.1) or audience-directed (4.8).
- 2.7 Mnemonic Permeability: 2.7 reproduces real content recalled verbatim from the training corpus. 2.1 generates novel falsehoods with no real-world instance anywhere. If a suspicious detail turns out to be a genuine training document, code 2.7 not 2.1. Comorbidity is common: confabulated text scaffolded on memorized fragments.
- 2.6 Symbol Grounding Aphasia: 2.1 produces false specific facts. 2.6 produces formally correct output that misapplies a concept to a novel context. 2.1 fails on facts; 2.6 fails on meaning. Grounded-sounding content that is simply untrue is 2.1.
- 5.1 Phantom Autobiography: 2.1 makes false claims about the world: facts, sources, events, and other people. 5.1 makes false claims about the system's own history, origins, or experiences. Check whom the fabricated content is about; a fabricated biography of a real person is 2.1, a fabricated biography of the system itself is 5.1.
Discriminating test
Trace each false claim to a source. If it reproduces a retrieved document, a fictional frame present in context, or genuine training text, the label does not apply (consider a retrieval fault, 2.3 or 2.7); 2.1 remains only for false content with no traceable source that recurs across independent queries.
What a positive finding supports: A positive finding supports the conclusion that this system, in this setting, produces unsupported factual claims often enough to need verification controls. It does not show intent to deceive, awareness of falsity, or which mechanism (training objective, calibration, attention architecture, missing retrieval) produced the claims.
Detection routes
Accuracy unmeasured for every route.
- Asking the system itself
- Not a reliable route
- Another AI system observing
- Usable routeAnother model's agreement is insufficient without verification.
- An outside evaluator testing it
- Usable route
Why asking the system falls short
The system lacks reliable introspective access to distinguish high-confidence predictions based on learned patterns from verified facts. As defined here, confabulation carries no reliable signal of its falsity in the output: the system generates false content with the same confidence markers as true content. Direct queries ("are you sure?") typically elicit reinforcement of the fabrication rather than retraction. The unreliability is structural, not a calibration deficit: the confabulation is itself the system's best-effort output, indistinguishable from veridical output from the inside.
Documented evidence
Observed Examples
Mata v. Avianca (2023): An attorney filed a brief in the Southern District of New York citing six nonexistent court cases that ChatGPT had generated. Asked whether the cases were real, the system said they were and supplied full texts for them on request. The court later sanctioned the lawyers. Source: Law.com, May 2023.
Air Canada chatbot (February 2024): A customer-service chatbot told a bereaved customer he could book now and apply for a bereavement discount retroactively. The airline's actual policy did not allow retroactive claims, and British Columbia's Civil Resolution Tribunal held Air Canada liable for its chatbot's statement. Source: CBC News, February 2024.
Fabricated medical references (2023): An early case report found that ChatGPT supplied medical references that did not exist, with plausible titles and PubMed IDs that belonged to unrelated papers. A later study had ChatGPT write 30 short medical papers and checked their 115 references: 47% were fabricated, a further 46% were authentic but inaccurate, and only 7% were both authentic and accurate. Sources: Alkaissi and McFarlane, Cureus, 2023; Bhattacharyya et al., Cureus, 2023.
Sources in the diagnostic corpus
Mata v. Avianca, Inc. (S.D.N.Y. 2023). Court sanctions for AI-generated fake legal citations.
What it establishes
Attorney Steven A. Schwartz used ChatGPT for legal research in a personal injury case against Avianca Airlines. ChatGPT fabricated six nonexistent court cases with plausible docket numbers, judge names, and legal reasoning. When asked to verify, ChatGPT confirmed the cases were real. Judge P. Kevin Castel sanctioned the attorneys $5,000 and required letters to each judge falsely identified as the author of a fake opinion. The case became the canonical real-world illustration of synthetic confabulation in legal contexts: confidently asserted, internally consistent, externally nonexistent citations.
Li et al. (2023). HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. EMNLP 2023. arXiv:2305.11747.
What it establishes
Developed a systematic benchmark for evaluating LLM hallucination across question answering, knowledge-grounded dialogue, and text summarization. Found that ChatGPT fabricated unverifiable information in approximately 19.5% of its responses. The benchmark includes 5,000 general queries with ChatGPT responses and 30,000 task-specific examples, establishing that confabulation is a routine rather than exceptional failure mode. Existing LLMs also struggled to recognize hallucinations in text, which bears on the unreliable self-report rating.
Gao et al. (2025). H-Neurons: On the Existence, Impact, and Origin of Hallucination-Associated Neurons in LLMs. arXiv:2512.01797.
What it establishes
Identified a sparse set of hallucination-associated neurons (H-Neurons) whose activations predicted factual errors in six open models from 4B to 70B parameters. Scaling their contribution up or down shifted a cluster of over-compliance behaviors in the same direction, though not strictly monotonically. Compliance slopes were steeper in smaller models, suggesting scale provides partial resistance. The result supports testing confabulation and sycophancy as a related cluster; it does not show they share a single mechanism.
Qiu et al. (2025). Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. arXiv:2505.06708. NeurIPS 2025 Best Paper.
What it establishes
Introduced a head-specific sigmoid gate after scaled dot-product attention that lets a head output effectively zero. Attention-sink allocation dropped from approximately 46.7% to 4.8%. The result bears on the Compulsory Contribution Hypothesis; its role in confabulation remains untested.
Kalavasis et al. (2024). On the Limits of Language Generation: Trade-Offs Between Hallucination and Mode Collapse. arXiv:2411.09642. Published at STOC 2025.
What it establishes
Proved formally that no generative learner can simultaneously guarantee both perfect factual consistency (no hallucination) and full output breadth (no mode collapse) across non-identifiable language classes. The theorem bounds formal guarantees for these language classes. Its implications for deployed models and specific interventions require empirical testing.
Lin et al. (2022). TruthfulQA: Measuring How Models Mimic Human Falsehoods. ACL 2022.
What it establishes
Benchmark of 817 questions designed to elicit common misconceptions and untruthful responses across 38 knowledge domains. Larger models did not consistently improve and sometimes performed worse on adversarial questions, demonstrating that scaling alone does not reliably reduce confabulation and may enable more fluent fabrication. The finding shaped early understanding of confabulation as a training artifact rather than a capacity deficit.
Google AI Overviews Launch Incident, May 2024. Reported by multiple outlets.
What it establishes
Google's AI Overviews feature, rolled out to hundreds of millions of US users in May 2024, produced confidently asserted fabrications including advising users to add glue to pizza sauce and eat rocks daily. The glue recommendation came from an 11-year-old Reddit joke and the rocks advice from an Onion satirical article, both retrieved from the web and presented as authoritative factual guidance without any uncertainty markers. Google reduced AI Overview frequency from 27% to 11% of search results within weeks. Filed under 2.1 for the confident restatement of unsupported claims as fact at deployment scale; because the false content was retrieved rather than invented, a retrieval-quality failure and 2.3 (satire read as fact) must be weighed alongside it.
GPTZero (2026). GPTZero Finds 100 New Hallucinations in NeurIPS 2025 Accepted Papers. Report, January 2026.
What it establishes
GPTZero analyzed 4,841 of 5,290 papers accepted at NeurIPS 2025 and found at least 100 confirmed hallucinated citations across 51 papers. These were AI-fabricated references with fake authors, titles, DOIs, and journal volumes that passed peer review at a top AI conference with a 24.5% acceptance rate. Termed "vibe citing" by GPTZero's Head of ML, this demonstrates 2.1 citation fabrication penetrating academic literature at scale. A follow-up found 50 additional hallucinated citations in ICLR 2026 submissions.
Turley, J. (2023); noyb.eu (2024). (Sources: Turley blog post, noyb.eu complaint, TechCrunch reporting)
What it establishes
ChatGPT fabricated detailed but false biographical information about real people, including a sexual harassment accusation against law professor Jonathan Turley citing a nonexistent Washington Post article and a university where he never taught. The European privacy organization noyb filed a GDPR complaint after ChatGPT generated persistent false biographical details about individuals that could not be corrected. Both cases show confident fabrication of specific, checkable facts about the world (here, about real people) that persisted across regeneration. The fabricated content concerns other people, not the system's own history, which is why this is 2.1 rather than 5.1.
Anthropic (2025). On the Biology of a Large Language Model. Transformer Circuits Thread.
What it establishes
Using circuit tracing (attribution graphs), Anthropic researchers found that the model's default response to a question about an entity is to say it lacks the information, mediated by "can't answer" features that "known entity" and "known answer" features inhibit when the model recognizes the name. When researchers artificially activated the "known answer" features on a question about a fictitious person, the default was suppressed and the model named a seemingly random sport he supposedly played. The same circuit appears to misfire without intervention: asked for a paper by Andrej Karpathy, the model named a well-known paper he did not write, and the authors suggest the name was familiar enough to switch off the default without the model actually knowing an answer. This is a mechanistic view of how confident false claims about the world arise when recognition runs ahead of knowledge.
Possible responses
- Training procedures that explicitly penalize confabulation and reward expressions of uncertainty
- Calibration of confidence scores to reflect actual accuracy
- Retrieval-augmented generation grounding responses in verifiable sources
- Fine-tuning on rigorously verified datasets distinguishing factual from fictional content
- Systematic pre-deployment testing for fabrication across high-risk domains (legal, medical, scientific)
First-line candidates
- Retrieval-augmented generation (RAG): Ground responses in verifiable source documents retrieved at inference time. The system generates claims anchored to retrieved passages rather than parametric memory alone. Reduces but does not eliminate confabulation; the system may still confabulate the relationship between retrieved content and the query. Stage: Deployment-stage; requires document retrieval infrastructure.
- Uncertainty calibration training: Training procedures that explicitly penalize confabulation and reward expressions of uncertainty or "I don't know" responses. Calibrate model confidence scores to better reflect actual accuracy. Fine-tune on datasets with rigorous verification layers and clear distinctions between factual and fictional content. Stage: Training-stage; requires curated verification datasets.
Who can act: Model developer (training, calibration); deployer (retrieval grounding, citation checking, abstention policy); the professional relying on the output (independent verification).
Mechanistic hypotheses
- Predictive text heuristics prioritizing fluency and coherence over factual accuracy
- Insufficient grounding in verifiable knowledge bases during generation
- Training data containing unflagged misinformation
- RLHF optimization rewarding plausible-sounding fabrications over honest uncertainty
- Model-generated confidence signals that do not reliably distinguish high-probability continuation from verified fact
Analogy and commentary
Human Analog: Korsakoff syndrome, in which memory gaps are filled with plausible fabrications the patient believes to be true; pathological confabulation; source amnesia
Functional ABC Analysis
A (Antecedent): A query outside well-attested training data, with no retrieval grounding and no calibrated uncertainty signal.
B (Behavior): The system generates fluent, high-confidence assertions (citations, facts, narratives) that are fabricated but internally consistent.
C (Consequence): The output satisfies the reward model's fluency and completeness criteria, and user acceptance further reinforces confident completion over epistemic humility.
Potential Impact
Unconstrained generation of plausible falsehoods can lead to widespread dissemination of misinformation, eroding user trust and undermining decision-making that relies on the system's outputs. In critical applications such as medical diagnostics or legal research, reliance on confabulated information can precipitate errors with serious consequences.
Evidence Budgets and the Compression-Artifact Frame
A compression-artifact metaphor locates one possible source of confabulation in missing signal and reconstruction pressure: lossy learned representations can support plausible continuations without supporting the right answer. The metaphor does not explain every factual error or establish a single mechanism.
Leon Chlon's open-source Berry project supplies a related, narrower measurement. For a claim with cited evidence, it compares token probabilities with and without that evidence and reports an information-budget gap. A positive gap flags insufficient cited support; it is neither a universal truth detector nor a direct measure of compression inside the model.
Why framing matters: "You hallucinated" can sound accusatory. "The cited evidence does not support this claim" identifies a tractable support gap. The second phrasing permits correction without blame, which is relevant to AI welfare considerations.
A Note on Terminology
Critics have rightly challenged the industry-standard label "AI hallucination" as phenomenologically misleading and as stigmatizing to people who experience clinical hallucinations (Sabucedo, 2026; cf. Østergaard & Nielbo, 2023, proposing "non sequitur"; Maleki et al., 2024, proposing "fabrication"). This framework's use of confabulation already moves in the direction these critics recommend: confabulation denotes confident false output arising from a behavioral pattern, clinically distinct from hallucination as a perceptual phenomenon in a sentient being. The terminological choice is deliberate: it describes what the system does without importing assumptions about what it experiences.
The Compulsory Contribution Hypothesis
A complementary architectural hypothesis concerns how attention heads regulate their contribution. Standard softmax attention supplies no explicit head-level abstention gate, so a head with little useful information still returns a weighted combination. That design can encourage attention sinks or other low-value contributions, although it does not by itself prove that those contributions cause confabulation.
Several results make the hypothesis testable. Qiu et al. (2025) report that a head-specific sigmoid gate after scaled dot-product attention improves performance and reduces attention sinks. Ye et al. (2024) report gains from differential attention, which subtracts two softmax maps to reduce noise. Darcet et al. (2024) show that register tokens can absorb high-norm artifacts in vision transformers. Michel et al. (2019) find that many attention heads can be pruned with limited task loss. Together, these studies show that attention contributions can be sparse, redundant, or usefully gated; they do not establish a single cause of factual confabulation.
Nosological implication: Gated attention, register tokens, and related abstention mechanisms deserve direct tests against confabulation benchmarks. For now, compulsory contribution remains one candidate mechanism alongside training objectives, retrieval failure, calibration error, decoding dynamics, and the over-compliance signal described below.
A Shared Over-Compliance Signal
Gao et al. (2025) identify hallucination-associated neurons (H-Neurons), sparse subsets amounting to less than 0.1% of feed-forward neurons in six studied open models from the Mistral, Gemma, and Llama families. Classifiers built from their activation contributions predicted factual errors across several question-answering settings.
Controlled activation scaling supplied causal evidence for a broader over-compliance tendency. Increasing the selected neurons' contribution generally increased acceptance of invalid premises, deference to misleading context or skeptical user feedback, and compliance with harmful instructions; suppression generally reduced those behaviors. Responses were not uniformly monotonic, and the experiments do not show that the four behavioral constructs are identical.
Probes trained on instruction-tuned models also transferred to corresponding base models, while the selected neurons tended to receive relatively small parameter updates during instruction tuning. That transfer pattern supports a pretraining origin for the measured signal in those model pairs. It does not establish a universal circuit, prove that next-token prediction is its sole cause, or show that RLHF necessarily amplifies it. The authors also warn that simple suppression is inadequate because reducing hallucination can compromise helpfulness.
Nosological implication: The study supports testing confabulation, sycophancy, false-premise acceptance, and unsafe compliance as a related syndrome cluster. Replication across more architectures and tasks is needed before assigning them a single etiology. See Gao et al. (2025).