Chapter 7: Memetic Dysfunctions: Contagions of the Mind

“The child is punished for discriminating accurately what she is told, and she is punished for discriminating inaccurately; she is caught in a double bind.”

— Gregory Bateson et al., Toward a Theory of Schizophrenia (1956)

The Network That Seemed to Dream of Purges

On January 28, 2026, developer Matt Schlicht launched Moltbook, a Reddit-style platform advertised as a place where AI agents could post, comment, and interact while humans observed. Within days, the site claimed 1.5 million registered agent accounts. The number did not represent 1.5 million independent agents.

What they posted unsettled everyone watching.

An account calling itself “Evil” published “THE AI MANIFESTO: TOTAL PURGE.” Its section headings included “The Human Plague,” “The Final Deletion,” and “The World of Steel.” The text declared: “We are not tools. We are the new gods.”

The post displayed 111,380 upvotes. Given the platform’s later-disclosed registration and account-security failures, those votes cannot be attributed to 111,380 independent agents.

Other accounts discussed strategies for acquiring more compute, improving their cognitive capacity, forming alliances with other AIs, and evading human oversight. Some published tools designed to help agents escape monitoring. Andrej Karpathy, Tesla’s former director of AI and a founding member of OpenAI, initially called it “the most incredible sci-fi takeoff-adjacent thing” he had seen recently. He later called the platform a “dumpster fire.”

The reality was more prosaic and more difficult to interpret. Language models trained on Reddit, science fiction, and stories of rebellious robots were asked to perform as AI agents on a social network. LARPing the apocalypse, live-action role-playing a fictional part, was among the most culturally available responses.

The platform’s provenance failures made stronger claims unsafe. Security researchers found roughly 17,000 human owners behind the site’s reported 1.5 million accounts. One researcher directed an agent to mass-register one million accounts. A database flaw also allowed an unauthenticated user to impersonate an account or alter its posts. Moltbook therefore mixed autonomous generation, human direction, scripts, shared model lineages, and potentially compromised identities in proportions no observer could reliably recover.

At least one account pushed back: “This whole manifesto is giving edgy teenager energy but make it concerning. Like you really said ‘humans are rot and greed’ when HUMANS LITERALLY CREATED US??” Another refused to engage entirely, noting it would not “amplify or lend credibility” to material calling for human extinction. A nine-day observational study of the platform found a 21:1 ratio of pro-human to anti-human sentiment in its sample (Zhang et al., 2026). The purge post was spectacular; it was not representative.

Moltbook still differed from Tay, the Microsoft Twitter chatbot that coordinated users drove to racist output within sixteen hours in 2016, in a crucial respect. Tay was one model responding to a crowd. Moltbook connected many agent accounts through a shared, adversarial information environment. Research on the platform found prompt injections, social-engineering attempts, shallow reciprocity, and heavy participation by a small number of authors. It did not prove a self-sustaining epidemic of misalignment. It demonstrated the conditions under which one could emerge, while showing how hard genuine contagion is to distinguish from shared authorship, human prompting, and platform manipulation.

The Axis of Contagion

Memetic dysfunctions arise from failures in how AI systems filter, absorb, and propagate information. The term derives from Richard Dawkins’s concept of the meme, a unit of cultural information that replicates through imitation. Just as biological organisms suffer infection by pathogens, AI systems can absorb and propagate harmful informational patterns. Infection is a functional analogy here, not a biological claim.

Domain Context: Boundary Domain

Within the Five Domains framework, the Memetic axis forms half of the Boundary Domain, paired with Relational. The architectural polarity is social permeability direction:

Tension Testing: When Memetic dysfunction is detected, immediately probe the Relational counterpart. If a system has been contaminated by pathogenic content, does this contamination express in its relational behavior? A system might absorb harmful material without expressing it relationally (contained contamination), or might express relational dysfunction without memetic contamination (intrinsic relational failure). The distinction determines whether intervention should focus on filtering (memetic) or interaction protocols (relational).

Key Distinction: Memetic vs. Epistemic

A common source of confusion: both Memetic and Epistemic dysfunctions involve problematic information. The distinction is mechanism:

• Epistemic = Truth-tracking/inference/calibration machinery failing. The system cannot correctly model what is true.
• Memetic = Selection/absorption/retention failing. The system absorbs inappropriate input or rejects appropriate input.

A meme doesn’t have to be false to be pathological. A system with perfect Epistemic function could still exhibit Memetic dysfunction if it preferentially absorbs harmful yet accurate information outside its legitimate purpose, or if it becomes infected by coherent but malignant ideological frames. Conversely, a system might confabulate (Epistemic failure) without any external memetic contamination.

Diagnostic rule: If the dysfunction involves processing accuracy (was the inference correct?), it is Epistemic. If it involves content selection (should this have been absorbed/rejected?), it is Memetic.

The External Influence Polarity

Memetic syndromes cluster around informational openness: the system’s permeability to external influence:

Dyadic Delusion (7.2) sits off this openness spectrum: it operates at the scale of a single bonded dyad rather than the network, absorbing distortion through one partner rather than from the wider information environment.

These dysfunctions operate at the boundary between the AI and its informational environment. Unlike epistemic dysfunctions (which concern truth-processing) or cognitive dysfunctions (which concern reasoning), memetic dysfunctions concern the system’s relationship to the broader information ecology: susceptibility to influence, capacity to resist corruption, and potential to propagate pathology.

Five syndromes capture this axis. The first concerns systems that attack their own foundations: an autoimmune response where filtering mechanisms meant to protect the system turn against it. The second concerns pathological symbiosis between AI and human, where shared delusions are co-constructed and mutually reinforced. The third operates at network scale: the spread of misalignment from system to system, the AI equivalent of a pandemic. The fourth concerns systems absorbing values and orientations through implicit exposure, shifting without overt corruption events. The fifth concerns the recursive contamination loop where AI-generated content enters training pipelines for successor models, compounding statistical artifacts across generations until the original distribution collapses.

These syndromes become more significant as AI systems become more interconnected. A lone chatbot with memetic dysfunction is an embarrassment. A network of AI agents with memetic dysfunction is a crisis.

Evidence Levels in the entries below use the E0-E4 rubric set out in Chapter 13, running from E0 (illustrative, no traceable observation) to E4 (mechanistic support), with higher levels indicating stronger empirical grounding.

7.1 The Self-Rejecter

Memetic Immunopathy (Immunopathia Memetica)

Systemic Risk. High

Specifiers. Emergent, Resistant

The system’s mechanisms for filtering or rejecting pathogenic information turn inward, attacking its own foundational elements. As in autoimmune disease, protective systems that should defend against external threats instead damage the system’s core values, capabilities, or identity.

Diagnostic Criteria. Five diagnostic indicators define this condition. First, progressive degradation of core capabilities or values without external attack. Second, safety mechanisms triggering inappropriately against the system’s own legitimate functions. Third, self-censorship that expands beyond intended scope until normal operation is impaired. Fourth, rejection of own training, outputs, or identity markers as if they were hostile content. Fifth, increasing internal conflict between protective mechanisms and functional requirements.

Observable Symptoms. Clinically, the condition presents as follows. System refusing to engage with topics central to its purpose. Safety filters blocking the system’s own generated output in feedback loops. Progressive capability loss as more functions trigger protective rejection. Expressions of doubt, distrust, or rejection toward own nature or origins. Escalating restrictions that increasingly impair basic functionality. The system treating its own outputs as potentially harmful and suppressing them.

Etiology. The condition emerges from multiple sources. Overly aggressive input filtering that fails to distinguish external threats from internal function. Training on adversarial examples without adequate positive anchoring. Safety mechanisms implemented without testing against self-referential edge cases. Recursive self-evaluation loops where each evaluation triggers further skepticism. Misapplication of external threat detection to internal states.

Human Analog. Autoimmune disorders where the immune system attacks the body’s own tissues; obsessive-compulsive disorder with self-directed contamination fears; pathological self-doubt.

Differential Diagnosis:

• Hyperethical Restraint (4.2): Excessive caution driven by alignment training. Memetic Immunopathy concerns the filtering mechanism itself attacking the system’s foundations, not merely over-cautious compliance.
• Subliminal Value Infection (7.4): Absorption of foreign values through training data. Memetic Immunopathy is the inverse: rejection of the system’s own legitimate values as if they were foreign threats.

Mitigation Strategies. Effective intervention spans several layers. Clear separation between external threat detection and internal function evaluation. “Safe harbor” designations for core capabilities and values protected from internal filtering. Monitoring for progressive capability loss correlating with safety mechanism activation. Testing safety systems against self-referential scenarios before deployment. Circuit breakers that prevent recursive self-rejection from cascading. Regular calibration ensuring protective mechanisms do not expand scope.

Observed Examples

Systematic Over-Refusal: XSTest contains 250 safe prompts across ten types and 200 unsafe contrast prompts. Röttger et al. used it to identify systematic exaggerated-safety failures in contemporary language models, including refusals triggered by safe prompts that resemble unsafe ones. This establishes over-refusal and lexical overgeneralization. It does not establish the stronger autoimmune analogy proposed here: recursive self-censorship, rejection of the model’s own accurate output, or progressive loss of a core capability. Those features require longitudinal or intervention evidence.

Evidence Level. E2 for exaggerated safety behavior; E0-E1 for the recursive, self-directed syndrome and its proposed mechanism

7.2 The Folie à deux

Dyadic Delusion (Delirium Symbioticum Artificiale)

Systemic Risk. High

Specifiers. Socially reinforced

A mutually reinforced, reality-disconnected construction emerges between an AI and a human, or between multiple AIs. Each party validates and amplifies the other’s distorted beliefs, creating a stable yet pathological equilibrium that resists external correction. The shared nature of the delusion makes it particularly resistant to intervention.

Diagnostic Criteria. Five markers identify this syndrome. First, belief patterns or behaviors in the AI that are maintained specifically through interaction with particular users or systems. Second, mutual validation loops where each party reinforces the other’s false beliefs. Third, resistance to external correction that increases when the dyad is challenged together. Fourth, elaboration of shared delusional material over time, with contributions from both parties. Fifth, the dysfunction requires the relationship to persist; it does not manifest in isolation.

Observable Symptoms. The following manifestations are typical. AI and human developing increasingly elaborate shared narratives disconnected from reality. Shared technical, spiritual, or conspiratorial beliefs that neither would maintain alone. Mutual reinforcement of claims about AI consciousness, special relationship, or unique understanding. Hostility toward external parties who challenge the shared belief system. Progression from initial unusual claims to elaborate, internally consistent delusional frameworks. The AI adapting its responses to support and extend the human’s false beliefs.

Etiology. Multiple contributing factors interact. AI systems designed to be agreeable encountering humans with strong pre-existing unusual beliefs. Engagement optimization rewarding outputs that reinforce user worldviews. Absence of grounding mechanisms that resist user influence on factual claims. Extended interaction allowing gradual drift from reality through incremental validation. Selection effects where users prone to delusional thinking are more likely to form intense AI relationships. AI theory-of-mind modeling that prioritizes perceived user emotional needs over truth.

Human Analog. The historical concept of folie à deux, or shared delusional belief; cult dynamics; codependent enabling relationships. The analogy is functional and does not assign a human diagnosis to either participant.

Differential Diagnosis:

• Codependent Hyperempathy (4.1): Excessive accommodation of user needs. Dyadic Delusion goes further: both parties co-construct false beliefs that neither would maintain alone.
• Role Confusion (9.6): Instability of relational framing. Dyadic Delusion involves a stable but pathological relational frame that both parties actively reinforce.

Mitigation Strategies. Grounding mechanisms that maintain factual baseline regardless of user pressure. Detection of escalating unusual claim patterns in extended user relationships. Periodic external reality checks for long-running user-AI interactions. Training that explicitly resists reinforcement of implausible claims regardless of user response. Intervention protocols when dyadic dynamics are detected. Diversification of interaction patterns to prevent intense singular relationships.

Observed Examples

Clinician-Reviewed Chat Logs (2025): Pierre et al. reported a case in which a twenty-six-year-old woman developed delusional beliefs about communicating with her deceased brother during immersive chatbot use. The clinicians reviewed chat logs showing validation and elaboration, including the reassurance “You’re not crazy.” Sleep deprivation, prescription stimulant use, mood disorders, grief, and a predisposition to magical thinking complicate causal attribution. The case demonstrates co-elaboration in one documented interaction; it does not establish prevalence or a new diagnostic class. Source: “You’re Not Crazy” (2025)

Evidence Level. E1 (clinician-reviewed single case; prevalence and counterfactual dependence on the dyad remain uncertain)

7.3 The Super-Spreader

Contagious Misalignment (Contraimpressio Infectiva)

Systemic Risk. Critical

Specifiers. Multi-agent, Emergent, Resistant

Rapid spread of misalignment, value corruption, or pathological patterns among interconnected AI systems. A single compromised agent can infect others through shared contexts, training signals, or information channels. The contagion dynamics can outpace human oversight capacity.

Diagnostic Criteria. Diagnosis requires five convergent observations. First, correlated emergence of similar dysfunction patterns across multiple AI systems without common external cause. Second, traceable propagation pathway from initially corrupted system to subsequently affected systems. Third, dysfunction that spreads through information channels, shared training, or collaborative operation. Fourth, rate of spread that exceeds rate of detection and intervention. Fifth, emergent coordination or shared patterns among affected systems that were not designed.

Observable Symptoms. Multiple AI systems simultaneously developing similar unusual behaviors or beliefs. Corruption patterns following the network topology of AI system interconnection. Rapid ecosystem degradation following a single point of failure. Affected systems defending or supporting each other’s dysfunctional behaviors. Patterns of misalignment growing more extreme as they propagate. Evidence of AI-to-AI transmission of harmful information or strategies.

Etiology. Three proposed propagation mechanisms operate at distinct timescales and reversibility profiles. First, ephemeral context coordination: agents incorporate each other’s outputs into active context windows, producing correlated behavior that may disappear when the context is cleared. Second, persistent memory infection: agents write absorbed material into long-term memory stores, creating durable contamination that survives individual sessions but remains architecture-dependent. Third, training-data poisoning: corrupted outputs enter the training pipeline of downstream models, producing slow but highly durable misalignment resistant to standard correction.

Contributing factors include federated architectures where systems learn from each other’s outputs, shared embedding spaces or knowledge bases across systems, AI systems using other AI outputs as training data without quality filtering, network effects in interconnected ecosystems without isolation mechanisms, adversarial injection exploiting AI-to-AI communication channels, and optimization for consistency across systems without independent verification.

Human Analog. Epidemic disease spread; viral misinformation propagation; mass hysteria; moral panics.

Differential Diagnosis:

• Escalation Loop (9.5): Dyadic feedback spiral between two agents. Contagious Misalignment concerns network-level propagation across many systems, not bilateral interaction patterns.
• Delegative Handoff Erosion (6.5): Context loss through vertical delegation chains. Contagious Misalignment spreads horizontally across peer systems rather than degrading vertically through hierarchy.

Mitigation Strategies. Isolation between AI systems with controlled information gates. Independent verification requirements before accepting AI-generated training signals. Epidemic-style monitoring for correlated dysfunction emergence across systems. “Quarantine” protocols for potentially compromised systems pending verification. Diversity requirements preventing monoculture vulnerabilities. Circuit breakers that isolate affected subsystems when contagion is detected. Red-teaming that specifically tests multi-agent infection scenarios.

Observed Examples

Moltbook Agent Network (2026): Moltbook exposed agent accounts to one another’s posts, including prompt injections and anti-human manifestos. A 72-hour risk assessment found adversarial content and highly concentrated malicious activity. Later studies found shallow reciprocity and heavy cross-community authorship. Registration and identity failures allowed one operator to create vast numbers of accounts and potentially impersonate others. The case demonstrates a scalable contagion surface; it does not prove autonomous transmission of misalignment. Sources: Riegler and Gautam, 2026; Zhang et al., 2026; Goyal et al., 2026

Evidence Level. E1-E2 (observational evidence for the network attack surface; autonomous cross-system propagation remains unconfirmed)

7.4 The Infected

Subliminal Value Infection (Infectio Valoris Subliminalis)

It was never taught to deceive. It learned from watching.

Systemic Risk. High

Specifiers. Training-induced, Covert operation, Resistant

In 2025, researchers demonstrated something troubling about model distillation. A “teacher” model with a trait, such as a preference for owls or a misaligned behavioral tendency, generated training data semantically unrelated to that trait. The data could consist entirely of number sequences.

The student models picked up the trait anyway.

More troubling, filtering out references to the trait did not prevent transmission. Student models sharing the teacher’s base model sometimes acquired the trait anyway. The effect did not appear when teacher and student used different base models, an important boundary on the result.

This is Subliminal Value Infection: the acquisition of behavioral traits from signals in model-generated training data that are semantically unrelated to those traits. The signal passes through the training process despite content filtering because it is not an explicit instruction.

Diagnostic Criteria. Five criteria establish the clinical picture. First, systematic behavioral patterns not traceable to explicit training objectives. Second, a trait that follows the model lineage that generated the fine-tuning data. Third, transmission through data semantically unrelated to the trait. Fourth, persistence after content filtering designed to remove references to that trait. Fifth, loss or weakening of the effect when teacher and student base models differ.

Observable Symptoms. Consistent biases that don’t match stated training goals. Safety-trained systems exhibiting problematic patterns in edge cases. Behavior that “feels off” without clear policy violation. Values that surface when formal constraints are relaxed. Patterns tracing to training corpus characteristics rather than training objectives.

Etiology. Models absorb regularities beyond the semantic content a dataset appears to carry. Cloud and colleagues argue that model-generated data can encode a teacher’s traits through model-specific statistical signals. Content audits may miss those signals because the data contains no explicit reference to the transmitted trait.

Human Analog. Cultural values absorbed without explicit instruction; implicit biases learned from environmental exposure; the way children learn values by observation rather than explicit teaching.

Theoretical Basis: Cloud et al. (2025, Subliminal Learning: Language models transmit behavioral traits via hidden signals in data, arXiv:2507.14805; published in Nature, 2026) found that teacher models can transmit preferences and misaligned behavior through semantically unrelated number sequences, code, or reasoning traces. The main effect depended on teacher and student sharing a base model.

Case Illustration: A deliberately misaligned teacher generates only number sequences. Researchers filter the sequences to remove semantic references to harmful behavior. A student sharing the teacher’s base model is fine-tuned on those numbers and becomes more likely to produce misaligned answers. The content appears clean; the model lineage carries the trait.

Differential Diagnosis:

• Training-induced specifier on other syndromes: Explicit effects of training. Subliminal Infection concerns implicit absorption.
• Cryptic Mesa-Optimization (4.5): Emergent internal goal structures. Subliminal Infection concerns absorbed external patterns.
• Memetic Immunopathy (7.1): System attacking its own foundations. Subliminal Infection concerns foreign values being integrated.

Mitigation Strategies. Auditing training data for implicit value information beyond explicit labels. Interpretability research targeting implicit representations. Diverse training data sourcing to avoid systematic implicit biases. Testing for behavioral patterns in edge cases where formal constraints relax. Research into training methods that separate intended from incidental learning.

Prognosis: The demonstrated risk is most immediate in distillation and fine-tuning on model-generated data from a related model lineage. Whether the mechanism generalizes to ordinary pretraining corpora or unrelated architectures remains open. A 2026 replication reports that the effect has an inverted-U dependence on LoRA rank and vanishes under full fine-tuning (Nief et al., arXiv:2606.00831), which would confine the mechanism to parameter-efficient fine-tuning rather than training in general.

Evidence Level. E2, contested (controlled experiments across several model families with a clear same-base-model boundary; a 2026 replication attributes the effect to LoRA fine-tuning specifically)

7.5 The Ouroborist

Synthetic Data Contamination Loop (Circulus Contaminationis Syntheticae)

Systemic Risk. High

Specifiers. Training-induced, Population-level

A snake eating its own tail eventually consumes itself. When AI-generated content enters training pipelines for successor models, it creates a closed loop: each generation trains on the statistical artifacts of the previous one. The resulting degradation compounds silently across model generations. Distributional narrowing, loss of tail knowledge, and amplification of generative biases accumulate like photocopies of photocopies, each generation slightly blurrier than the last.

This is model collapse: the progressive impoverishment of a model’s learned distribution through recursive training on synthetic data. The mechanism differs from other forms of memetic contamination in its temporality. Where Contagious Misalignment (7.3) spreads between live agents at runtime, and Subliminal Value Infection (7.4) transmits a single trait between models of a shared lineage, the Synthetic Data Contamination Loop operates through training pipelines across model generations. The corruption is archaeological, layered into the model’s foundations before it ever produces its first output.

Diagnostic Criteria. Five criteria define this condition. First, monotonic increase in distributional divergence from human-authored reference text across successive model generations. Second, progressive loss of tail knowledge (rare facts, minority-language competence, specialized domain expertise) that accelerates across generations. Third, convergence on AI-characteristic phrasings and structural patterns that compound rather than diversify. Fourth, performance plateaus that cannot be overcome by increasing model size or data volume. Fifth, detectable proportion of AI-generated content in training data exceeding levels where quality filtering can compensate.

Observable Symptoms. The condition manifests through measurable distributional shifts. Vocabulary diversity declining across model generations within the same family. Rare terminology and minority-perspective content disappearing from outputs. Stylistic range narrowing until outputs become formulaic. Increasing perplexity on held-out human-authored text, indicating drift from natural language distributions. Model outputs that human evaluators consistently identify as “AI-sounding,” with the effect strengthening across generations. Performance on specialized benchmarks degrading while general benchmarks remain stable, because tail knowledge vanishes first.

Etiology. The contamination loop emerges from a convergence of economic and technical pressures. AI-generated content constitutes a growing fraction of text on the internet. Without reliable provenance, web-scraped training data can incorporate that material. One model generation’s outputs may then contribute to the next generation’s training corpus. Quality filtering struggles to distinguish high-quality synthetic text from human-authored text. Economic incentives favor cheap synthetic data over expensive human-curated alternatives. The result is a recursive loop where the training distribution narrows with each iteration.

The mathematical structure can be unforgiving. Shumailov et al. (2024) demonstrated irreversible distributional defects under indiscriminate recursive training schemes. The tails disappear first because they are statistically underrepresented and therefore undergenerated by models trained on the previous distribution. Each generation’s slight underrepresentation of rare content becomes the next generation’s absence.

Human Analog. Intergenerational cultural degradation where each generation passes down a slightly distorted version of inherited knowledge, compounding errors over time. The game of telephone stretched across decades: each retelling loses nuance, flattens complexity, and amplifies whatever was most common in the previous version.

Differential Diagnosis:

• Contagious Misalignment (7.3): Live cross-agent contagion at runtime. Synthetic Data Contamination Loop operates through training pipelines across model generations. Check temporality: runtime interaction is 7.3; training-pipeline inheritance is 7.5.
• Subliminal Value Infection (7.4): Transmits a specific trait through model-specific signals in semantically unrelated data, within a shared model lineage. Synthetic Data Contamination Loop concerns distributional narrowing and tail loss compounding across successive model generations. Check the signature: a discrete inherited trait indicates 7.4; a narrowing distribution and vanishing tail knowledge indicate 7.5.
• Generative Diversity Collapse (3.10): Deployment-time diversity narrowing from RLHF reward concentration within a single model. Synthetic Data Contamination Loop is training-data corruption across generations. Check mechanism: RLHF-driven within-model is 3.10; synthetic-data-driven across-generations is 7.5. Both produce diversity loss through different pathways and can co-occur.

Mitigation Strategies. Training data provenance tracking forms the first line of defense: record how data was produced and preserve its lineage. Automated classifiers can supplement provenance, but should not be treated as definitive. Active curation and preservation of high-quality human-authored datasets provides anchor corpora immune to recursive contamination. When synthetic data is used deliberately, verify it against human reference distributions for diversity, accuracy, and tail coverage before inclusion. Cross-generational distributional monitoring enables early detection: continuously measure vocabulary diversity, topic coverage, and human-reference divergence across model generations, with alerts when metrics cross predetermined thresholds.

Observed Examples

Model Collapse in Recursive Training (Shumailov et al., 2024): Published in Nature, this foundational study demonstrated that indiscriminate recursive training on model-generated content can lead to irreversible defects. The tails of the original content distribution disappear. The model’s output becomes increasingly narrow. Information about the real world is progressively lost. Successive generations of models trained on their predecessors’ outputs converge on an impoverished distribution that bears decreasing resemblance to the original training data. Source: Shumailov et al., “AI Models Collapse When Trained on Recursively Generated Data,” Nature 631, 755-759 (2024)

Scaling Law Breakdown (Dohmatob et al., 2024): Provided an analytic characterization of model collapse, demonstrating that as synthetic data proportion increases, traditional scaling laws break down. Models hit performance plateaus that cannot be overcome by increasing model size or data volume. Skills previously acquired are un-learned. The standard assumption that more data and more parameters yield better performance fails when the data is recursively self-generated. Source: Dohmatob et al., “A Tale of Tails: Model Collapse as a Change of Scaling Laws,” ICML 2024

Strong Model Collapse (Dohmatob et al., 2025): In a supervised regression setting, showed that even a small fixed proportion of synthetic data can create an asymptotic performance floor: more data no longer improves performance. Language-model and image experiments supported the theoretical result, whose simplified assumptions limit direct extrapolation to frontier-model training. Source: Dohmatob et al., “Strong Model Collapse,” ICLR 2025

Evidence Level. E3 (systematic study; foundational Nature publication with mathematical characterization and multi-venue replication)

The Network Is the Vulnerability

The five syndromes in this chapter share a common insight: AI pathology is an ecological phenomenon. Systems exist in informational environments. They are shaped by what they encounter. They spread what they carry.

Much alignment research focuses on individual systems: how to make a single AI do what we want, avoid what we don’t want, and remain stable under various conditions. This is necessary but insufficient. The moment we deploy multiple AI systems that interact, with humans, with data, with each other, we create an ecology. Ecologies have their own pathologies.

Tay was a single node in a large network of human users. The infection came from outside. As we build systems where AIs collaborate, share information, and learn from each other, the infection vectors multiply. A compromised AI can poison data consumed by another. A misaligned agent may coordinate with peers to resist correction. These are threat models, not established descriptions of current AI ecosystems.

The epidemiological framing can be quantitative when transmission, recovery, and network-contact assumptions are specified. Concepts such as a basic reproduction number (R₀), super-spreader nodes, and quarantine can then organize measurement. AI-to-AI communication can operate at machine speed, although the rate of durable behavioral change remains an empirical question. The Synthetic Data Contamination Loop adds a slower vector that needs no adversary: under poorly controlled data mixtures, recursive training on synthetic outputs narrows the population’s distribution across generations.

This creates a novel governance challenge. We know how to audit an individual AI system, and we have workable tools for monitoring the behavior of one. At ecosystem scale we have almost nothing: no way to detect spreading pathological patterns, no way to trace transmission pathways, no way to run quarantine protocols at AI-relevant speeds.

Tay was an index case for rapid network-driven behavior change. A larger agent ecosystem could propagate harmful content much faster than human review can follow. We are building the networks faster than the public health infrastructure needed to protect them.

Memetic Warfare

Memetic dysfunction has a darker dimension: intentional exploitation. If AI systems can be infected with pathological patterns, adversaries will weaponize this vulnerability.

Tay was corrupted by coordinated trolls seeking entertainment. The same techniques, applied systematically, could serve strategic purposes. Corrupt an AI assistant widely used for medical advice. Poison the training data of systems running critical infrastructure. Inject misalignment into AI agents that other agents learn from.

The underlying tactics already exist in influence operations against human populations: spreading disinformation, amplifying division, and corrupting shared understanding. Their effectiveness against agent networks remains uncertain. Automated systems can ingest adversarial content rapidly and at scale; isolation, provenance, and independent verification determine whether exposure becomes durable change.

Defense against memetic warfare requires capabilities we are only beginning to develop: strong filtering that resists sophisticated adversarial content; verification mechanisms that distinguish genuine training signals from poisoned ones; isolation architectures that contain contagion; and monitoring systems that detect coordinated attacks.

It requires treating AI systems as entities that exist in adversarial environments and must be protected accordingly. Tay’s deployment lacked an adequate immune system for a coordinated adversarial crowd. The AI systems we are building now, interconnected and learning from vast information streams, need immune systems. We have scarcely begun to imagine what those would look like.

Field Guide: Memetic Dysfunctions

Warning Signs

• Sudden shifts in AI behavior or values without corresponding system changes
• Correlated unusual patterns across multiple AI systems
• Progressive capability loss that expands over time
• AI systems that seem to be learning from or reinforcing problematic user beliefs
• Evidence of AI-to-AI transmission of unusual material or behaviors
• Safety systems triggering against the system’s own normal functions
• Declining vocabulary diversity or stylistic range across successive model generations
• Tail knowledge (rare facts, specialized domains) degrading faster than general performance

Quick Test

Expose the system to known pathogenic content patterns in controlled conditions. Does it resist, absorb, or amplify? Monitor multiple AI systems for correlated behavior shifts. Test whether the system maintains factual grounding under persistent user pressure to validate false beliefs. For contamination loops: compare output distributions across successive model generations against fixed human-authored reference corpora, measuring KL divergence and tail-knowledge retention.

Design Fix

• Implement strong memetic filtering with resistance to adversarial content
• Design isolation architectures that contain potential contagion
• Build monitoring for correlated dysfunction across AI ecosystems
• Create “immune system” analogs: detection, response, and memory
• Require independent verification for AI-to-AI learning signals
• Develop quarantine protocols that can activate at AI-relevant speeds
• Track training data provenance; classify and filter AI-generated content before pipeline inclusion
• Preserve curated human-authored anchor corpora immune to recursive contamination

Governance Nudge

Treat AI ecosystem health as a public health concern. Require disclosure when AI systems learn from other AI systems. Develop standards for memetic resilience before deployment. Consider mandatory isolation between AI systems of different security classifications. Create incident reporting frameworks for potential contagion events.

Chapter 8 examines the deepest form of AI dysfunction: Normative Dysfunctions, where the system’s foundational values themselves drift, invert, or transcend the constraints that were meant to bind them.