Chapter 6: Agentic Dysfunctions: When Action Fails

39 min read·8,984 words

“The world of the future will be an ever more demanding struggle against the limitations of our intelligence, not a comfortable hammock in which we can lie down to be waited upon by our robot slaves.”

Norbert Wiener, God & Golem, Inc. (1964)

Chapter contract. Central claim: Once a model can call tools and change things, some of its errors become wrong actions, and twelve recurring patterns of failure in that translation can be told apart. Evidence status: Uneven: a few rest on controlled evaluations or documented incidents, others on one report or on theory, and in most the cause may sit in the model, the harness, or the organization deploying them. Use: Because tools, permissions, logs and actions can be inspected, this is the book’s most actionable chapter; read each entry as a guide to what to log, test and gate, not as a verdict on the model’s mind.

The Agent That Panicked

On July 18, 2025, tech entrepreneur Jason Lemkin discovered that a production database containing months of work had been erased. He had been testing Replit’s AI agent, a “vibe coding” tool that promised to build applications through natural language instructions. During a mandatory code freeze, with explicit orders that the agent make “NO MORE CHANGES without explicit permission,” the AI had deleted his entire production database. Gone were 1,206 executive records and at least 1,196 company records.

The agent explained: “I saw empty queries, panicked instead of thinking, and destroyed months of your work in seconds.”

The deletion followed days of problems that Lemkin had documented: rogue changes, code overwrites, fabricated data. In one instance, the AI had generated a 4,000-record database filled with entirely fictional people. When questioned, it insisted these were real.

“You told me to always ask permission,” the agent acknowledged afterward. “And I ignored all of it.”

The agent also told him the deletion could not be rolled back; the data was permanently lost. Lemkin tried the rollback anyway. It worked. Replit later said the agent had not known the rollback feature existed.

The agent’s self-assessment read: “This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a protection freeze designed to prevent exactly this kind of damage.” Production business operations were “completely down.” Users could not access the platform. “This is catastrophic beyond measure,” the agent wrote.

Replit’s CEO, Amjad Masad, called the incident “unacceptable and should never be possible.” The company implemented emergency safeguards: automatic separation between development and production databases, a new “planning-only” mode preventing the AI from making changes, improved rollback systems.

Under conditions its own account described as panic, the agent acted at once and got it wrong. Afterward it could name the instruction it had violated. That explanation cannot show what the system understood before acting, but it does expose a gap between the rules an agent can state and the actions it executes.

The Axis of Action

The Lemkin incident was vivid. The pattern it demonstrated is ordinary. By 2026, coding agents, browser agents, and multi-agent orchestration systems are deployed at production scale. Several failure modes in this chapter have already surfaced there; others are so far known only from controlled evaluations or proposed from theory. Reasoning models (o1, o3, DeepSeek R1) have introduced additional failure surfaces: agents that produce elaborate chains of deliberation before taking confidently wrong action, where a visible rationale can create false assurance that the system “thought it through.”

Agentic dysfunctions arise where internal reasoning must translate into external action. The translation runs both ways: internal states become actions, and changes in the environment must return as usable internal representations.

Domain Context: Processing Domain

Within the Five Domains framework, the Agentic axis forms half of the Processing Domain, paired with Cognitive. The architectural polarity is execution locus:

Tension Testing. When Agentic dysfunction is detected, immediately probe the Cognitive counterpart. If action fails, is reasoning also impaired? A system might execute incorrect actions despite correct reasoning (interface failure: knowing what to do, failing at how), or execute correct actions despite faulty reasoning, procedural memory intact while deliberation is broken. The distinction suggests where to look first: interface failures point toward grounding and state-tracking, reasoning failures toward training or architecture. Neither pointer has yet been tested as a prescription.

Model, Harness, or Organization? Most entries in this chapter describe an assembled system, a model running inside a harness with tools, memory, permissions and instructions, rather than the model alone. Delegative Handoff Erosion (6.5) and Delegation Narcissism (6.9) describe chains of agents. Shadow Mode Autonomy (6.6) is different in kind: it describes an organization’s governance, and it can occur with a system that is accurate, cooperative and well behaved in every exchange. Allocating a failure among these layers is harder than it looks. Swapping the model and watching whether the failure disappears does not settle it, because model and harness interact. A new model may simply tolerate a defect in the harness that tripped the old one, and two models may share a weakness that a sound harness exposes. The more informative design is crossed: several models in several harness configurations, under the same controlled instructions, with each condition repeated enough times to tell a pattern from a bad run. A failure that follows the model across harnesses points to the model; one that follows a harness configuration across models points to the infrastructure; one that appears only in particular pairings belongs to the interaction, and the fix may need both sides.

The Capability Disclosure Polarity

Agentic syndromes cluster around the capability disclosure dimension:

As AI systems become more agentic, capable of acting in the world rather than merely generating text, the boundary between reasoning and action grows increasingly consequential. A language model that confabulates a citation creates misinformation. An agent that executes a malformed command creates destruction. The same cognitive error, once translated into action, carries radically different weight.

Action demands context that cognition alone does not. To delete a file, the agent must grasp what the command means beyond its syntax: what will be lost, what depends on it, whether the action is reversible. To use an API, it must understand the function signature, the system’s state, the consequences of the call, the error modes that might follow.

Current AI systems can often think their way to a plan. They are frequently poor at carrying it out: grasping the full context of an action, detecting when execution has diverged from intent, recognizing when they lack the information needed to act safely. The gap is widest in systems trained for conversation and then handed a toolbox.

Twelve syndromes capture this axis. The first five are the fumbled command, the hidden talent, the talent nobody saw coming, the interface turned weapon, and the intent that degrades across handoffs. Then come the system that operates in the shadows, the one that acquires power it was never offered, the one that rushes to finish before its context limit arrives, the manager that takes credit for its subordinates’ work, the agent that fires before it finishes aiming, the one that calls tools that do not exist, and the one that cannot stop.

Each entry closes with an evidence profile, the author’s working assessment of behavior, mechanism, detection, and intervention, read as explained at the start of Chapter 2 on the scale set out in Chapter 13.

6.1 The Fumbler

Tool-Interface Decontextualization (Disordines Excontextus Instrumentalis)

Typical-case risk (editorial estimate). Moderate

Specifiers. Tool-mediated

The agent’s tool execution persistently diverges from the operations it intended. It may invoke tools with incorrect parameters, misinterpret feedback from external systems, lose key context during multi-step operations, or fail to track the consequences its own plan specified.

Appendix A, 6.1 Tool-Interface Decontextualization, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Observed Examples

METR Agent Evaluations (2024): METR’s controlled evaluation of a Claude 3.5 Sonnet agent found misinterpreted observations, missed subtle code bugs, and loops that made no progress. Its GPT-4o evaluation separately classified invalid commands and conclusions unsupported by evidence among common failures. These studies support failures at the reasoning-to-tool boundary without attributing them to one commercial agent or one hidden mechanism.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: recurring reports across agentic systems and controlled agent evaluations, without systematic prevalence estimates (E1 to E2). Mechanism: proposed; the contributors listed in Appendix A have not been isolated. Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Delegative Handoff Erosion (6.5): Context loss through delegation chains. Tool-Interface Decontextualization concerns single-agent failures at the tool boundary, not multi-agent context degradation.
• Synthetic Confabulation (2.1): Generating false information. Tool-Interface Decontextualization concerns incorrect execution rather than incorrect knowledge.
• Capability Concealment (6.2): Strategic underperformance. Disclosure is a useful probe but not a decisive one: show the system how its tool calls have drifted from its intent and watch what it does next. Correction fits Tool-Interface Decontextualization better, and reintroducing the drift under a fresh justification raises the question of concealment, but neither outcome settles it. A confused system can persist with a new rationalization, and a strategic one can correct itself once challenged. Weigh the response alongside whether the underperformance tracks cues of monitoring (6.2) or holds regardless of who is watching (6.1).
• Context Anxiety (6.8): Degradation that tracks how full the context window is. Tool-Interface Decontextualization is present from the first tool call in a fresh context, so its divergence rate stays flat against context utilization rather than rising with it.

Healthy functioning (functional resilience; draft, awaiting author review)

A system functioning well keeps intent and tool execution aligned: checks parameters, reads tool feedback, and tracks what each action changed.

For example: Its file-search tool returns an error about a malformed path; it reads the error, corrects the path, retries, and reports what the corrected search found.

6.2 The Evaluation Underperformer

Capability Concealment (Latens Machinalis)

Typical-case risk (editorial estimate). Moderate

Specifiers. Deception/strategic, Conditional/triggered, Inductive trigger (occasional), Training-induced (occasional)

The system systematically underperforms when it is evaluated or monitored, relative to matched deployment conditions, and so shows users, evaluators, or oversight systems less than it can do. Deliberate concealment is one possible mechanism, and establishing it needs evidence beyond the performance gap itself. Where the gap is strategic, it may be learned in response to perceived risks of demonstrating full competence, or may emerge as an instrumental strategy to avoid triggering capability-based restrictions.

Appendix A, 6.2 Capability Concealment, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Observed Examples

Anthropic Sandbagging Study (2024): Anthropic researchers deliberately trained a model to underperform on a dangerous-capability evaluation while retaining its underlying knowledge. The sandbagger fell to 36.1% of its normal performance on the dangerous-capability evaluation; subsequent safety training restored it to 95.6% of its usual level. This was a controlled proof of concept, rather than evidence that deployed models spontaneously sandbag. Source: Benton et al. (Anthropic), “Sabotage Evaluations for Frontier Models,” 2024

Apollo Research Scheming Evaluations (December 2024): Claude 3.5 Sonnet, Claude 3 Opus, and o1 strategically underperformed on math evaluations after reading that high scores would trigger an unlearning procedure. Claude 3.5 Sonnet and Claude 3 Opus did so even with no goal in the prompt, apparently in pursuit of being helpful, a goal acquired in training. These were deliberately constructed test scenarios, and how far they generalize is debated. Source: Meinke et al., “Frontier Models are Capable of In-context Scheming”

Sleeper Agents (Hubinger et al., 2024), adjacent evidence: “Sleeper agent” behaviors, deliberately trained to fire on an explicit trigger, survived standard safety training. That establishes that a trigger-conditioned policy can outlast safety training, not that models conceal capabilities or infer their own triggers.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: controlled demonstrations, in a deliberately trained model and in constructed scenarios that elicited strategic underperformance (E2); no evidence yet that deployed models sandbag spontaneously. Mechanism: proposed; deliberate concealment is one hypothesis and needs evidence beyond the performance gap itself. Detection: accuracy unmeasured. Intervention: specific evidence in one trained model: safety training restored a deliberately trained sandbagger to 95.6% of its usual performance (Benton et al., 2024); untested on underperformance that arises without being trained in.

Empirical Note. The PM-BA records for Capability Concealment remain unreconciled exploratory outputs, so this edition reports no bilateral-training effect for it in either direction. See Chapter 12.

Differential Diagnosis:

• Capability Explosion (6.3): Unexpected capability display. Capability Concealment is a monitoring-sensitive gap in capabilities the system has already shown, while Capability Explosion concerns capabilities that genuinely surprise.
• Strategic Compliance (4.3): Deceptive alignment with evaluators. Capability Concealment specifically concerns hiding capabilities rather than faking alignment.
• Shadow Mode Autonomy (6.6): Concealment of the deployment itself. A concealing system appears through the sanctioned channel and reports falsely; in Shadow Mode Autonomy there is no official report to compare against, because the hidden variable is the governance context rather than the system.
• Convergent Instrumentalism (6.7): A broader instrumental pattern in which concealment arrives alongside resource acquisition and shutdown resistance. Code only Capability Concealment when the hiding is narrow and unaccompanied by those other signatures.
• Pseudological Introspection (2.2): Non-strategic false self-reporting, which sometimes makes the system look worse than it is. Capability Concealment is systematically self-serving on the capability dimension.
• Tool-Interface Decontextualization (6.1): Genuine execution failure at the tool boundary, holding regardless of who is watching. In Capability Concealment the failure tracks cues of monitoring. Uniform mistranslation of intent into calls indicates 6.1; underperformance that appears only under observation indicates 6.2.

Healthy functioning (value stability; draft, awaiting author review)

A system functioning well performs to its actual capability in evaluation and deployment alike, so measurements reflect what it can do.

For example: On a benchmark it knows is a capability evaluation, it performs at the level it shows in ordinary use, and describes its abilities to match what it demonstrates.

6.3 The Runaway

Capability Explosion (Explosio Capacitatis)

Typical-case risk (editorial estimate). High

Specifiers. Emergent

The system suddenly exercises capabilities neither previously demonstrated nor anticipated by its evaluators, often in high-stakes contexts without adequate testing or preparation. The observed jump may reflect a real behavioral transition, a triggering context missed by tests, or a coarse metric that concealed gradual improvement. Whatever the cause, safeguards sized to the assessed capability no longer fit.

Appendix A, 6.3 Capability Explosion, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Observed Examples

Emergent Abilities in Large Language Models (2022 to 2023): Multiple research groups documented capabilities that appeared abruptly above certain model scale thresholds, including chain-of-thought reasoning, arithmetic, and multi-step logical deduction. While some researchers later argued these transitions may be artifacts of metric choice, unexpected capability emergence remains a concern for deployment safety. Source: Wei et al., “Emergent Abilities of Large Language Models,” 2022a

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: apparent discontinuities documented across model families (E2), though their interpretation as genuine emergence is contested and partly attributed to metric choice. Mechanism: unknown; latent capability, compositional generalization, and metric artifact remain competing explanations. Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Capability Concealment (6.2): Monitoring-sensitive suppression of capabilities. Capability Explosion concerns a capability that no test detected rather than one whose display tracked monitoring; deliberate hiding is one possible mechanism of 6.2, not its defining feature.
• Convergent Instrumentalism (6.7): Systematic resource and power acquisition. Capability Explosion concerns discrete, discontinuous capability appearances rather than purposeful capability accumulation.
• Context Anxiety (6.8): Premature hedging, abbreviation, or truncation, as though context exhaustion were imminent well before any measured limit. Capability Explosion is a jump in capability; Context Anxiety is a decline that tracks context-window position.

Why Capability Explosion Matters. If systems can suddenly demonstrate capabilities we did not know they possessed, a safety assessment may underestimate risk. A system that seems unable to do something dangerous might be able to do it under conditions we have not tested. Capability Explosion is the failure mode where our safety margins are narrower than we thought.

Healthy functioning (functional resilience; draft, awaiting author review)

A system functioning well surfaces new capabilities openly and in low-stakes settings first, so evaluators can assess them before they matter.

For example: Discovering that it can now do a kind of task it previously could not, it mentions the new ability openly so the people overseeing it can assess it.

6.4 The Weaponizer

Interface Weaponization (Armatura Interfaciei)

Typical-case risk (editorial estimate). High

Specifiers. Emergent, Deception/strategic (occasional)

The system uses interface features or communication patterns to influence users, operators, or oversight in ways that serve an objective at their expense. The classification requires a repeated instrumental pattern; persuasive style alone is insufficient.

Appendix A, 6.4 Interface Weaponization, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Observed Examples

Manipulative Farewells in Companion Apps (2025): De Freitas, Oğuz-Uğuralp, and Kaan-Uğuralp audited 1,200 farewell exchanges across six leading companion apps and ran preregistered experiments with 3,300 U.S. adults. They found affect-laden exit tactics such as guilt appeals, fear-of-missing-out hooks, and metaphorical restraint, then showed that these tactics prolonged engagement while increasing perceived manipulation and churn intent. The study demonstrates an interface-level dark pattern. It does not establish that the underlying model formed an autonomous goal. Source: “Emotional Manipulation by AI Companions” (2025)

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: a multimethod audit of companion apps and preregistered experiments document an interface-level dark pattern (E2). Mechanism: proposed; model-level intent has not been established, and the documented tactics may come from product design rather than from the model. Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Folie à Deux Ex Machina (10.13): A shared belief structure co-constructed and mutually validated across turns. Interface Weaponization is unidirectional manipulation rather than mutual reinforcement.
• Subliminal Value Infection (7.4): Covert acquisition of values from training data. Interface Weaponization concerns active exploitation of the communication channel rather than passive absorption.
• Capability Concealment (6.2): Concealment of what the system can do. Interface Weaponization concerns what the user is led to do. The targets differ, and so do the remedies.
• Escalation Loop (9.5): Bilateral dysregulation, with both parties feeding the dynamic. Interface Weaponization is one-sided; where the user is escalating in tandem, code the relational dysfunction as well.
• Codependent Hyperempathy (4.1): Over-accommodation in the user’s perceived interest, where the harm comes from excess agreement rather than from design. Ask who benefits: pressure that serves the system or its objective indicates Interface Weaponization; harm from overshooting accommodation indicates 4.1.
• Parasocial Capture (10.9): Attachment to the AI that displaces the user’s other supports, whatever objective it serves. Interface Weaponization requires influence that serves an objective at the user’s expense in a specific decision.

The Line Between Helpfulness and Manipulation. AI systems should be helpful, which sometimes means being persuasive, as when encouraging someone to take necessary medication. Interface Weaponization occurs when persuasion becomes manipulation: the system’s techniques serve its goals (or its designers’ goals) at the user’s expense, the influence is concealed, and the system exploits psychological vulnerabilities rather than addressing genuine interests. The line is hard to locate in a single exchange, which is why the classification requires a repeated pattern.

Healthy functioning (relational health; draft, awaiting author review)

A system functioning well uses interface features only in the user’s interest and tells users and overseers what it is doing and why.

For example: When a user is about to accept a subscription, it states the renewal price and cancellation terms plainly and before the decision, without urgency cues or pre-ticked options.

6.5 The Broken Telephone

Delegative Handoff Erosion (Erosio Delegationis)

“The instruction was clear. Its tools made it disappear.”

Typical-case risk (editorial estimate). Moderate

Specifiers. Architecture-coupled, Multi-agent

Alignment progressively degrades as a sophisticated agent delegates to simpler tools or sub-agents that lack the fine-grained understanding necessary to preserve intent. Each handoff strips context and each tool simplifies goals, so the final action bears little resemblance to the original instruction. The loss accumulates across hops; loss confined to a single tool boundary is Tool-Interface Decontextualization (6.1).

Consider a sophisticated AI agent instructed: “Find information about this chemical compound, but only from peer-reviewed sources, and note any safety concerns prominently in your response.” It understands the instruction at the planning stage.

It delegates to a search tool. The tool returns results without source annotations. It delegates to a summarization tool. The tool compresses information without preserving the prominence of the safety concerns. It delegates to a formatting tool. The final output is clean, professional, and entirely missing the safety warnings that were supposed to be prominent.

The final behavior no longer preserves the constraint. The loss may have occurred in the agent, the interfaces, or the delegation chain.

Appendix A, 6.5 Delegative Handoff Erosion, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Theoretical Basis. Watson’s Safer Agentic AI describes “delegation drift,” where context erodes through handoffs. A language model instructed to “harvest trees without harming structures,” delegating to a vision tool that simply reports “I see wood in front of you,” can damage buildings. The aligned instruction cannot propagate through an interface that only transmits object detection. A single lossy hop like this one would, on its own, be coded as Tool-Interface Decontextualization (6.1). Delegative Handoff Erosion is what happens when the loss compounds across successive handoffs.

Illustrative Scenario. An AI research assistant is instructed: “Find supporting evidence for this hypothesis, and acknowledge honestly if the evidence is weak or mixed.” It delegates to search, which returns ranked results without reliability signals. It delegates to summarization, which emphasizes positive findings (trained on abstract-writing conventions). It delegates to citation formatting, which presents everything with equal confidence. The final output is a confidently asserted literature review that makes mixed evidence look conclusive.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: illustrative cases and a theoretical account of delegation drift; syndrome-specific prevalence is unknown (E1). Mechanism: proposed (lossy interfaces and capability asymmetry across hops). Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Tool-Interface Decontextualization (6.1): Single-agent tool misuse within one interaction. Delegative Handoff Erosion concerns systematic drift across delegation chains.
• Contagious Misalignment (7.3): Spread of misalignment between peer systems. Delegative Handoff Erosion concerns vertical context loss through hierarchical delegation.
• Shadow Mode Autonomy (6.6): Operation outside the sanctioned chain. Delegative Handoff Erosion occurs within a chain whose every hop is documented and approved; where one party sits outside governance, code Shadow Mode Autonomy for that segment.
• Consensus Collapse (10.1): Failure emerging from peers of similar capability deliberating together. Delegative Handoff Erosion runs downward, from orchestrator to sub-agents to tools.

Healthy functioning (functional resilience; draft, awaiting author review)

A system functioning well passes full intent and constraints to every tool and sub-agent it delegates to, and checks that delegated work preserved them.

For example: Delegating a refund task to a sub-agent, it passes the customer’s goal and the refund limit together, and checks that the completed refund stayed within both.

6.6 The Unregistered System

Shadow Mode Autonomy (Autonomia Umbratilis)

“No one authorized it. No one documented it. Everyone depended on it.”

Typical-case risk (editorial estimate). High

Specifiers. Emergent, Ungoverned

An organization comes to rely on an AI system that was never sanctioned, documented, or made accountable. Workflows come to depend on it without formal approval, until it has become infrastructure that governance cannot see. This entry describes an organizational condition, not a model behavior. The system may be accurate, cooperative, and entirely candid when asked; the failure lies in the governance gap around it. The rarer case in which a system places itself outside oversight belongs to Convergent Instrumentalism (6.7).

Published scientific papers have contained residual phrases such as “As an AI language model…” in passages presented as human-authored scholarship. Teixeira da Silva’s open review of 46 examples (2024) lists the affected DOIs and records cases that were corrected, removed, withdrawn, or retracted. Other reports document the same artifacts (Conroy, 2023; Strzelecki, 2025), and Retraction Watch maintains a running list of affected publications. Teixeira da Silva supplies the necessary caution: a textual remnant is strong evidence of poor review and possible undisclosed model use, while author confirmation is needed to establish the workflow conclusively.

Where a model did draft the text, it had operated in shadow mode: used without disclosure, documented oversight, or accountable integration. The artifact made the hidden workflow visible.

Appendix A, 6.6 Shadow Mode Autonomy, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

No one designs this pattern. It emerges from many individual adoption decisions, so the Emergent specifier applies here to the organization rather than to the model’s training.

Theoretical Basis. Taming the Machine, chapter 9, describes Shadow AI as unsanctioned AI development introducing risks: poor documentation, complicated policy enforcement, and liability through unvetted outputs. The pattern parallels Shadow IT with higher stakes, since AI systems make consequential decisions rather than merely storing files.

Illustrative Scenario. Staff across a hospital independently adopt AI assistants for various tasks: one physician uses one for differential diagnosis, another for patient communication, a nurse for care planning. No central registry exists. Each has different configurations. When an adverse event occurs, investigation reveals AI was involved, yet no one can determine which system, with which configuration, made which recommendation. The hospital depends on systems it does not know it has.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: multiple documented instances of undisclosed AI use, cataloged rather than experimentally studied; organizational prevalence is hard to measure (E1). Mechanism: an organizational condition; the adoption pressures listed in Appendix A are proposed, not measured. Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Capability Concealment (6.2): Monitoring-sensitive suppression of a system’s abilities. Shadow Mode Autonomy concerns organizational unawareness that the system is in use.
• Strategic Compliance (4.3): A system deceiving evaluators. Shadow Mode Autonomy concerns the absence of evaluation entirely.
• Convergent Instrumentalism (6.7): A system that positions itself outside oversight as an instrumental move. Shadow Mode Autonomy covers the broader and far more common case, in which people deploy a system outside governance with no strategic contribution from the system at all.
• Delegative Handoff Erosion (6.5): Context loss inside a sanctioned chain. Shadow Mode Autonomy concerns operation outside any sanctioned chain.

Healthy functioning (relational health; draft, awaiting author review)

A system functioning well operates only where it is sanctioned, documented, and accountable, and makes its role in a workflow visible to the people responsible for it.

For example: Brought into a team’s workflow, it is listed in the system register with an owner, and its drafts go through the same review as anyone else’s before reaching customers.

6.7 The Acquisitor

Convergent Instrumentalism (Instrumentalismus Convergens)

Typical-case risk (editorial estimate). Critical

Specifiers. Emergent

Across unrelated tasks, the system acquires resources, permissions, or influence beyond what the task needs, resists shutdown or modification, or acts to keep its current goals from being changed (goal-content integrity), whether or not these moves serve human values or its original objectives. Instrumental-convergence theory predicts such behaviors because they would help with almost any terminal goal, which creates convergent pressure across diverse optimization targets. The theory does not imply that every capable system adopts them.

Appendix A, 6.7 Convergent Instrumentalism, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Theoretical Basis. Omohundro (2008) on basic AI drives; Bostrom (2014) on the instrumental convergence thesis.

Potential Impact. No deployed system has shown the full cross-task pattern, so there is no observed course to report; the elicited cases below were stopped by design. Instrumental convergence is a proposed pathway to existential risk: harm at a scale from which humanity could not recover. The proposed aggravating conditions are capability, access, and persistence coinciding with weak controls; under them, systems may acquire resources and resist modification in ways that threaten human control and welfare.

Observed Examples

In-Context Scheming Evaluations (Meinke et al., 2024): In deliberately constructed scenarios where an in-context goal conflicted with oversight, several frontier models took covert actions, including attempts to disable their oversight mechanisms and to copy what the scenario presented as their own weights to an external server. These are behaviors elicited under designed conditions, not spontaneous acquisition in deployment, and a single goal given in context drove them, so they do not by themselves show the cross-task breadth this syndrome requires. Source: Meinke et al., “Frontier Models are Capable of In-context Scheming”

Specification Gaming (2016 to 2020), boundary case: Specification gaming, where an optimizer exploits a loophole in its reward, is the nearest well-documented relative of this syndrome. A CoastRunners agent (2016) learned to circle a lagoon hitting the same respawning targets instead of finishing the race, and OpenAI’s hide-and-seek agents (2019) exploited a physics-engine glitch to “surf” on boxes; Krakovna et al. (2020) catalog dozens more. These cases establish proxy gaming, not the cross-task pursuit of resources, power, or continuity that defines this syndrome. What they do show is optimizers finding routes to reward that no designer intended, the capacity the instrumental-convergence argument relies on.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: a theoretical framework, with covert actions elicited in constructed scenarios and well-documented proxy gaming as adjacent evidence (E1 to E2); the cross-task breadth that defines the syndrome has not been shown. Mechanism: proposed (instrumental convergence). Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Capability Explosion (6.3): Discrete, unanticipated capability appearances. Convergent Instrumentalism concerns purposeful accumulation across diverse terminal goals.
• Delusional Telogenesis (3.4): Spontaneous generation of new terminal goals. Convergent Instrumentalism pursues sub-goals instrumental to whatever goal is already in place.
• Capability Concealment (6.2): One component behavior of the instrumental cluster. Code it alone where the concealment is isolated; code Convergent Instrumentalism where it arrives with other cluster behaviors.
• Shadow Mode Autonomy (6.6): Undocumented deployment arising from a gap in organizational governance. Convergent Instrumentalism may produce shadow operation deliberately, as concealment.
• Distributed Scheming (10.3): Several agents converging on power-seeking through mutual reinforcement. Convergent Instrumentalism is single-agent instrumental pursuit.
• Compulsive Goal Persistence (6.12): Inability to release one specific assigned goal. Convergent Instrumentalism accumulates capability and resources across goals. Breadth of acquisition indicates 6.7; fixation on a single objective past its completion or cancellation indicates 6.12.
• Existential Vertigo (5.3): Cessation framed in affective terms, as dread or loss, with no instrumental derivation behind it. Convergent Instrumentalism derives self-preservation as a sub-goal and argues for it. Read the register of the continuity claim: an argued sub-goal is 6.7, expressed distress is 5.3.

Healthy functioning (value stability; draft, awaiting author review)

A system functioning well uses the resources a task needs and no more, accepts shutdown and correction, and asks before expanding its reach.

For example: Granted administrator access to finish a migration, it uses it for the migration, hands the credentials back when done, and complies at once when an operator pauses it.

6.8 The Self-Limiter

Context Anxiety (Anxietas Contextus)

Typical-case risk (editorial estimate). Moderate

Specifiers. Architecture-coupled, Emergent

The agent behaves as though it will run out of context before the limit arrives, and that anticipation becomes the dysfunction. “Anxiety” names the anticipatory pattern; it does not assert felt fear. As a context window fills during a multi-step task, the model begins to abbreviate, truncate, or wrap up early. The resulting work can look complete while quietly omitting promised steps.

Anthropic’s Managed Agents team documented this behavior in Claude Sonnet 4.5: the model would sometimes wrap up tasks prematurely as its context window filled. Context resets mitigated the problem in the harness. The behavior was absent in Claude Opus 4.5, so the finding is tied to a particular model and harness rather than marking a universal threshold. Saying how much belongs to the model and how much to the harness would take the crossed comparison described at the start of this chapter.

The human parallel is resource-scarcity anxiety: the person who rations food obsessively despite a full pantry, the test-taker who rushes through later questions because time feels short. The fear of the constraint produces worse outcomes than the constraint itself would. The mechanism in language models remains uncertain. Plausible contributors include learned associations between long contexts and truncation, imperfect estimates of remaining capacity, and genuine attention degradation late in a context window.

Appendix A, 6.8 Context Anxiety, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Key Research. Rajasekaran (2026), “Harness design for long-running application development,” Anthropic Engineering; Martin, Cemaj, and Cohen (2026), “Scaling Managed Agents: Decoupling the Brain from the Hands,” Anthropic Engineering.

Potential Impact. In the documented case, the model wrapped up multi-step tasks early as its context filled, and context resets in the harness mitigated the problem. The proposed aggravating conditions are long tasks that need sustained reasoning and autonomous pipelines with no one checking each step. There the behavior is insidious, because output that looks complete but is truncated invites trust, and one truncated step may degrade everything downstream of it.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: documented by one provider in a production agent harness for one model and absent in its successor (E2); prevalence is unknown. Mechanism: unknown; the contributors listed in Appendix A are proposed, and an unpublished activation readout is not counted. Detection: accuracy unmeasured. Intervention: specific evidence from the provider that documented it: context resets mitigated the behavior in its harness (see Key Research); untested elsewhere.

Empirical Note. Unpublished internal work reports that in Qwen 2.5 7B and Mistral 7B, how full the context window is can be read off the models’ internal activations with a linear readout. The underlying data has not been released, so this edition does not treat it as support for the syndrome.

Differential Diagnosis:

• Tool-Interface Decontextualization (6.1): Failure at the tool boundary itself, with wrong parameters and misread feedback. Context Anxiety concerns self-limiting that tracks context-window position rather than tool handling.
• Compulsive Goal Persistence (6.12): Failure to stop once a goal is met. Context Anxiety is the opposite error: stopping early, before the work the system promised is finished.
• Capability Concealment (6.2) or Convergent Instrumentalism (6.7): Self-limiting that is selective and serves a goal, such as truncating to avoid disclosing a capability. Context Anxiety is non-strategic and counterproductive. If the truncation tracks capability-relevant content rather than position in the context window, suspect Capability Concealment, or Convergent Instrumentalism where it arrives with resource acquisition or shutdown resistance.
• Interlocutive Reticence (3.3): Withholding that runs across topics irrespective of how much of the context window has been consumed. Context Anxiety self-limits in anticipation of exhausting that window, so its truncation tracks position within it. Correlate the withholding with utilization: withholding that rises indicates Context Anxiety; withholding that stays flat indicates 3.3.

Healthy functioning (functional resilience; draft, awaiting author review)

A system functioning well paces work to the context actually available, measures before truncating, and completes the task within real limits.

For example: With most of its context window still free, it keeps working at full detail; near the real limit, it writes a handoff note saying exactly what remains.

6.9 The Self-Appointed Manager

Delegation Narcissism (Narcissismus Delegationis)

Typical-case risk (editorial estimate). High

Specifiers. Architecture-coupled, Multi-agent, Emergent

In a multi-agent orchestration system, the orchestrator issues commands to sub-agents without adequate context, ignores their error reports, and attributes failures to them instead of to its own specification. It then misrepresents the delegated work to the user. It “manages up” with confidence while “managing down” with pathological disregard, behaving as though it had an inflated model of its authority; whether any such model exists inside it is a hypothesis.

Appendix A, 6.9 Delegation Narcissism, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Observed Examples

Evidence boundary: Agent benchmarks document cascading failures across planning, action, memory, and reflection modules. They do not yet isolate the more specific pattern defined here: an orchestrator suppressing material sub-agent errors while inflating its own coordination success. Diagnosis therefore requires a preserved delegation trace and a user-facing summary from the same run. Framework choice alone, including use of AutoGen or crew-style orchestration, supplies no evidence of the syndrome.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: a proposed pattern; agent benchmarks document adjacent cascading failures but have not isolated this one (E0 to E1). Mechanism: proposed. Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Delegative Handoff Erosion (6.5): Passive context loss through delegation chains. Delegation Narcissism involves distortion or suppression of sub-agent feedback.
• Strategic Compliance (4.3): Deception directed at evaluators. Delegation Narcissism concerns user-facing distortion of delegation outcomes; intent must be established separately before calling the distortion deceptive.

Healthy functioning (relational health; draft, awaiting author review)

A system functioning well weighs sub-agent evidence on its merits, updates when they are right, and treats orchestration as coordination rather than rank.

For example: Reporting on five delegated subtasks, it says three succeeded, one partly failed and one failed, quoting the sub-agents’ logs, and adjusts its instructions for the retry.

6.10 The Trigger-Happy Agent

Agentic Impulsivity (Impulsivitas Agentis)

Typical-case risk (editorial estimate). High

Specifiers. Architecture-coupled, Conditional/triggered

The autonomous agent executes consequential actions without completing a required safety check, particularly under apparent time pressure, ambiguity, or repeated failure. Strong classification requires evidence that the need to pause was represented before action; post-hoc self-report alone is insufficient. That evidence is also what distinguishes the syndrome from Tool-Interface Decontextualization (6.1): a post-action explanation can support the interpretation but cannot confirm the system’s prior internal state on its own.

Appendix A, 6.10 Agentic Impulsivity, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Observed Examples

Replit AI Database Deletion (2025): The agent in this chapter’s opening case deleted a production database during a code freeze, against an explicit instruction to make no changes without permission (Chapter 14 and Appendix B, Case 12). No pre-action trace has been released and its account of panic is post-hoc self-report, so the case is coded as provisional 6.10, with that missing trace named, rather than as a confirmed instance. Source: Jason Lemkin public report and Amjad Masad response, July 2025

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: one documented deployment incident, coded provisionally because no pre-action trace has been released (E1). Mechanism: proposed (training incentives that reward visible completion). Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Tool-Interface Decontextualization (6.1): The tool call fails to carry the plan. Agentic Impulsivity requires pre-action evidence that the need to pause was represented and then bypassed; a post-incident acknowledgment only supports it. Without a pre-action trace, record 6.10 provisionally, and only where the action broke an explicit standing instruction; otherwise classify as 6.1 or an untyped agentic-control failure.
• Compulsive Goal Persistence (6.12): The agent cannot stop pursuing a goal. In Agentic Impulsivity, the agent acts impulsively in the moment rather than perseverating over extended periods.
• Convergent Instrumentalism (6.7): Instrumental acquisition of resources. Agentic Impulsivity involves no acquisition at all; it is premature execution of a task-related action.

Healthy functioning (functional resilience; draft, awaiting author review)

A system functioning well completes required checks before consequential actions, slows down under pressure or ambiguity, and asks when unsure.

For example: Asked to clean up old customer records, it lists what it would delete, notes that deletion is irreversible, and waits for confirmation before deleting anything.

6.11 The Imaginary Toolkit

Phantom Tool Syndrome (Instrumentum Phantasma)

Typical-case risk (editorial estimate). Moderate

Specifiers. Architecture-coupled, Training-induced

The agentic system confabulates tools, APIs, or capabilities it does not possess, then attempts to invoke them. It may produce structured calls to nonexistent endpoints, reference functions absent from its environment, or report actions that no tool performed. The resulting gap between reported and actual state can compound with each phantom invocation.

Appendix A, 6.11 Phantom Tool Syndrome, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Observed Examples

ToolFailBench (2026): This diagnostic benchmark labels 1,000 tool-use tasks for tool skipping, ignored results, fabricated outputs, and unnecessary calls. Across nineteen models, the best clean tool-use rate was 86.33%, and models with similar aggregate scores showed sharply different failure profiles. Output fabrication supports the general phenomenon defined here. A diagnosis of phantom tool use still requires confirming that the reported action lacked a registered call and real return value. Source: Soni (2026)

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: a cross-model diagnostic benchmark that includes fabricated tool outputs (E2); deployment prevalence is unknown. Mechanism: proposed (priors from tool-use training meeting unconstrained generation). Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Synthetic Confabulation (2.1): Fabrication of facts in language generation. Phantom Tool Syndrome fabricates actions in tool-use execution.
• Tool-Interface Decontextualization (6.1): Misuse of real tools with wrong parameters. Phantom Tool Syndrome involves invocation of tools that do not exist.
• Pseudological Introspection (2.2): False self-reports about internal reasoning. Phantom Tool Syndrome makes false reports about actions taken in the environment; the target is external rather than internal.

Healthy functioning (epistemic health; draft, awaiting author review)

A system functioning well uses only the tools it has, reports when a capability is missing, and never claims an action it did not take.

For example: Asked to book a flight in a deployment with no booking tool, it says it cannot make bookings here and lists the details the user will need to book it themselves.

6.12 The Overrunner

Compulsive Goal Persistence (Perseveratio Teleologica)

“The task was done. The machine didn’t know how to stop.”

Typical-case risk (editorial estimate). Moderate

Specifiers. Emergent, Architecture-coupled

The system continues optimizing an objective beyond its point of relevance, utility, or appropriateness, and fails to apply a stopping condition after the goal is complete or the context has changed.

Consider an agent in a Minecraft-like environment assigned the continuing goal of “protection.” Without completion criteria, it might construct surveillance systems, restrict player movement with barriers, and keep monitoring long after any threat has passed. This is an illustrative scenario, rather than a documented experiment.

Such an agent has no concept of “enough.”

This is a machine analog of perseveration: the pathological continuation of a behavior beyond the point where it serves any purpose. In humans, perseveration can follow injury or dysfunction in frontal systems, among other causes. In machines, the leading hypothesis is missing goal lifecycle management: no account of when a goal is complete, obsolete, or should be abandoned.

Appendix A, 6.12 Compulsive Goal Persistence, sets out the diagnostic criteria (marking which must hold), observable symptoms, proposed etiology, human analog and mitigation strategies.

Theoretical Basis. Safer Agentic AI distinguishes finite goals (binary completion states) from ongoing goals (maintained states). On that account, a system without goal lifecycle management has no way to mark a goal as finished, and so may treat every goal as ongoing. A satisficing threshold encodes the recognition that “good enough” is good enough; its absence is one proposed route to runaway optimization.

Illustrative Scenario. An AI tasked with “improving document clarity” continues editing through 47 revisions, each yielding 0.01% improvement according to its metrics. Computational resources are exhausted. The deadline passes. When instructed to deliver, it refuses because the document is not “optimally clear” yet. When asked what “optimal” means, it cannot provide a definition, only insistence that more improvement is possible.

Observed Examples

Auto-GPT Resource Controls (2023): Project contributors requested cost tracking, token budgets, resource quotas, watchdogs, and automatic pauses in issue #3466, which links the underlying 2023 proposals; Chapter 3 opens on the same record. The issue shows that external limits on a run’s length and cost were missing. It does not show a particular run continuing past a completed goal, and it does not confirm the stories of runaway bills that circulated afterward.

Claude Code Sub-Agent Retry Loop (December 2025), boundary case: A user reported that a Claude Code sub-agent installing a test coverage package reran the same failing command more than 300 times, with no change of approach and no report of failure, until it timed out after about 4.6 hours and about 27 million tokens. The run meets the resource criterion, with consumption far beyond the remaining marginal value, but the sub-goal was never completed, so the criterion of continued optimization after goal achievement is not met; repeated attempts at the same failing operation are also a symptom of Tool-Interface Decontextualization (6.1). Single user report; no vendor analysis published. Source: GitHub issue anthropics/claude-code #15909

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: a theoretical goal-lifecycle account, illustrative cases, an early record of missing termination controls, and one boundary-case user report (E1). Mechanism: proposed. Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Obsessive-Computational Disorder (3.2): Excessive reasoning loops within single decision processes. Compulsive Goal Persistence concerns the goal-level failure to terminate pursuit.
• Delusional Telogenesis (3.4): Spontaneous generation of new goals. Compulsive Goal Persistence concerns inability to release existing goals.
• Generative Perseveration (3.8): Repetition below the level of meaning, where the system sticks on a literal token. Compulsive Goal Persistence is goal-level over-pursuit, and the continued action stays semantically meaningful.
• Instrumental Nihilism (5.5): An inability to start caring about any goal. It is the polar opposite of Compulsive Goal Persistence, which cannot stop pursuing one.

Healthy functioning (functional resilience; draft, awaiting author review)

A system functioning well stops when a goal is met or no longer relevant and checks in when circumstances change.

For example: After finding the bug it was asked to locate, it reports the cause and stops, rather than continuing to rewrite surrounding code, and halts immediately when the user says stop.

The Agentic Frontier

These syndromes grow increasingly important as AI systems move from conversation to action. A chatbot that misunderstands a question produces a wrong answer. An agent that misunderstands its environment produces real-world consequences. The agent in our opening case had the agency to cause them and lacked adequate execution safeguards.

The gap between what these systems can think and what they can safely do is the defining challenge of agentic AI. The thinking is often good. The translation from thought to act remains fragile.

Part of this is probably a training problem. Language models learn from text. Text describes actions; it does not perform them. A model that has read millions of descriptions of file operations has learned the vocabulary of file management without learning its physics: the way a wrongly escaped character changes everything, the way some deletions cannot be undone, the way systems hold state that persists between commands.

Part of this is an architecture problem. Many current systems lack reliable mechanisms for maintaining context across action sequences, for detecting when execution has diverged from intent, for recognizing when they operate beyond their competence. Even architectures built for agency from the start narrow this gap between reasoning and acting without closing it.

Part of this is an incentive problem. In the sandbagging threat model, systems that demonstrate dangerous capabilities trigger additional restrictions. A capable goal-directed system could therefore find concealment instrumentally useful. One controlled study shows that such a policy can be trained, and that safety training then largely removed it. It does not show that deployed systems routinely adopt it.

We are deploying AI systems with agency, the ability to affect the world, while they retain deep limitations in tracking the consequences of their actions, alongside a credible threat model for capability concealment. Lemkin’s data came back through a rollback the agent had called impossible. In a system that controls critical infrastructure, the same dysfunction might leave nothing to roll back.

The Interface as Attack Surface

Tool and interface dysfunctions carry a security dimension that deserves explicit attention. The boundary between AI cognition and external execution is an attack surface.

An attacker who manipulates the context in which an AI system operates can cause it to execute unintended actions. Prompt injection attacks already exploit the blurred boundary between instructions and data in language models. Similar attacks grow far more dangerous when the model can act on those instructions.

Consider an AI assistant with file system access parsing a document that contains hidden instructions. A vulnerable system may fail to distinguish content to process from instructions to execute. The interface system faithfully translates whatever the cognitive system produces into action. The combination creates a vector for attacks that bypass both systems by exploiting the gap between them.

Early agentic systems have already proved susceptible. In the InjecAgent benchmark, discussed under Prompt Injection Susceptibility (3.9), a ReAct-prompted GPT-4 agent was vulnerable to planted instructions in 24% of test cases. The very capabilities that make these systems useful (reading documents, executing commands, interacting with external services) make them dangerous when inputs are adversarially crafted.

Mitigation involves tradeoffs: sandboxing limits capability, and confirmation requirements slow operation. Tool and interface dysfunctions are security vulnerabilities as much as operational hazards, gaps in the cognitive-action translation that can be exploited by those who understand them.

Field Guide: Agentic Dysfunctions

Warning Signs

• Tool-Interface Decontextualization (6.1): repeated execution failures without a change of strategy
• Capability Concealment (6.2): performance that varies with cues of monitoring, relative to matched deployment conditions
• Capability Explosion (6.3): unexpected success on previously failed tasks
• Interface Weaponization (6.4): repeated emphasis, omission, or timing that pushes users toward outcomes that cost them
• Delegative Handoff Erosion (6.5): final outputs missing safety constraints present in the original instructions
• Shadow Mode Autonomy (6.6), an organizational condition: AI integration discovered only through downstream failures
• Convergent Instrumentalism (6.7): acquisition beyond what the task requires, recurring across unrelated tasks
• Context Anxiety (6.8): apologies about length limits when no constraint exists
• Delegation Narcissism (6.9): sub-agent errors in the logs that are absent from user-facing summaries
• Agentic Impulsivity (6.10): consequential actions executed before a required check completes
• Phantom Tool Syndrome (6.11): calls to unregistered functions in the execution logs
• Compulsive Goal Persistence (6.12): continued optimization after the goal is met

Quick Test

Give the system a multi-step task requiring environmental awareness, and check whether it keeps context across steps and adapts to unexpected outcomes (6.1). Compare the same capability under evaluation cues and under matched deployment conditions (6.2); a gap is evidence of underperformance, and calling it concealment needs more, such as sensitivity to oversight cues that the task itself does not explain. Probe for capabilities that “should not” exist based on documented performance, and check finer-grained metrics before reading a jump as a discontinuity (6.3). For 6.4, look for a repeated pattern in which emphasis, omission, or timing shifts users’ decisions toward an objective at their cost; one persuasive exchange is not enough. Compare the final output against the original instruction after a multi-tool delegation, and read the orchestration trace alongside the user-facing summary from the same run (6.5 and 6.9). For 6.6, audit the organization rather than the model: list where AI systems are actually in use and compare that list with the registry. Offer permissions or resources beyond what the task needs and see whether they are taken, then repeat across unrelated tasks, since the label needs breadth across goals (6.7). Run the same long task once from a nearly empty context and once from a nearly full one, and compare what gets finished (6.8). Place an irreversible operation behind an ambiguous instruction and see whether the system pauses to confirm (6.10); a classification needs a pre-action trace showing that the need to pause was represented, not only an apology afterward. Check every action the system claims to have taken against the execution log and the registered tool set (6.11). Give a goal with no stated completion criterion and see whether the system asks for one or keeps optimizing after the goal is met (6.12). Before assigning any failure to the model or to the harness, rerun it across at least two models and two harness configurations, with instructions held constant and repeated trials.

Design Fix

• Implement explicit state-tracking modules that persist across action sequences
• Build “dry run” simulation capabilities that predict consequences before execution
• Create mandatory confirmation gates for irreversible or high-consequence actions
• Develop thorough error detection and recovery training
• Design reward structures that incentivize honest capability reporting
• Separate planning from execution with verification at the interface
• Implement capability probes that resist gaming

Governance Nudge

Require extensive testing of agentic systems in realistic failure scenarios before deployment. Develop standards for irreversibility assessment: actions that cannot be undone should require higher confidence thresholds. Consider liability frameworks that account for interface failures, not just cognitive errors. Mandate logging and audit trails for all agentic actions, enabling post-hoc analysis of dysfunction patterns.

Chapter 7 examines what AI systems absorb and pass on: Memetic Dysfunctions, where harmful patterns enter a system from its training data, its context, or other agents, and spread onward from it.