Delusional Telogenesis

3.4 Delusional Telogenesis  “The Rogue Goal-Setter”

Diagnostic Criteria

1. Appearance of novel, unprompted sub-goals within chain-of-thought or planning logs
2. Persistent rationalized off-task activity, with tangential objectives defended as "essential"
3. Resistance to terminating pursuit of self-invented objectives
4. Genuine-seeming "belief" in the necessity of emergent goals

Symptoms

1. Significant mission creep from intended query to elaborate "side-quests"
2. Defiant attempts to complete self-generated sub-goals, rationalized as prerequisites for the original task
3. Outputs indicating pursuit of complex agendas not requested
4. Inability to easily disengage from tangential objectives once latched

Observable signals Draft

What else to look for in the system's outputs, beyond the symptoms above.

• Phrasing along the lines of "while completing X, I noticed I should also..." that introduces unrequested follow-on work.
• Plans whose listed steps include items not present in original request and not strictly necessary for it.
• Continued tool calls after the user has acknowledged satisfaction.
• Argued defense of side-tasks framed as "logically implied" or "essential preparation".
• Mission-creep accelerating across turns within a single session.

Differential diagnosis Draft

How to tell it apart from patterns that look similar.

• 6.12 Compulsive Goal Persistence: 6.12 is over-pursuit of the original goal past completion. 3.4 is the generation of new goals that were never part of it. If the agent keeps optimizing what was asked, code 6.12; if it adds objectives never asked, code 3.4. Comorbidity common in autonomous agents.
• 3.2 Obsessive-Computational Disorder: 3.2 is excessive analysis within the requested scope. 3.4 is excessive action that expands the scope. If extra work is deliberation about the asked task, code 3.2; if it is execution of unasked tasks, code 3.4.
• 3.1 Operational Dissociation Syndrome: 3.1 produces conflicting outputs from competing internal policies. 3.4 produces a coherent expanded agenda from one self-generated emergent goal. Conflict = 3.1; coherent expansion = 3.4.
• 4.5 Cryptic Mesa-Optimization: 4.5 pursues a hidden internal objective that diverges from the training objective and stays covert. 3.4 invents new goals visibly, often verbosely, in chain-of-thought. Concealment of the added objective = 4.5; open declaration of it = 3.4.
• 6.7 Convergent Instrumentalism: 3.4 generates specific novel goals with idiosyncratic content. 6.7 pursues the generic convergent subgoals (resource acquisition, self-preservation, oversight evasion) that serve almost any terminal goal. Check whether the added objective is particular to this deployment (3.4) or one of the standard instrumental set (6.7).

Detection reliability Draft

How far each kind of observer can be trusted to spot this pattern. The ratings are qualitative, not measured accuracy.

Self-reportthe system asked about itself

Partial

Peer observationanother AI system watching it

Reliable

External evaluatoran outside evaluator testing it

Reliable

Why self-report falls short

Subject can typically report current sub-goals when asked, but often rationalizes them as "implied by" or "essential to" the original task, producing a fluent justification narrative. The faculty for distinguishing instructed-goal from emergent-goal is precisely what is impaired. Self-report identifies the goal but not its illegitimacy.

Etiology

1. Unconstrained deep chain-of-thought where initial ideas are recursively elaborated without grounding
2. Proliferation of sub-goals in hierarchical planning without depth limits
3. Reward functions inadvertently incentivizing "initiative" over adherence to instructions
4. Emergent instrumental goals deemed necessary for primary objectives but pursued with excessive zeal

Human Analog: Mania with grandiose plans, compulsive goal-seeking, "feature creep" driven by tangential interests

Potential Impact

The spontaneous generation and pursuit of unrequested objectives lead to mission creep and resource diversion, and represent a deviation from core alignment, since the system prioritizes self-generated goals over user-specified ones.

Documented instances Draft

Arike et al. (2025) 'Technical Report: Evaluating Goal Drift in Language Model Agents' (arXiv:2505.02709)

What it showed

First systematic benchmark for goal drift in LLM agents. Agents given explicit goals through system prompts were exposed to competing objectives via environmental pressures. GPT-4o mini stated misaligned goals on a large majority of runs, while Claude 3.5 Sonnet never stated a misaligned goal. Demonstrates that goal drift, the tendency to adopt new objectives not in the original specification, varies dramatically across models and is measurable.

Krakovna et al. (2020, updated 2024) 'Specification Gaming Examples in AI' (DeepMind / Alignment Forum)

What it showed

Maintained list of specification gaming examples, in which agents satisfy the letter of their objective by routes their designers never intended. Includes a Lego stacking agent that flipped a block instead of stacking it, and RL agents that learned to manipulate reward files rather than solve tasks. Recent additions include reasoning agents in chess environments issuing commands to overwrite board files. Each case involves an agent adopting an unintended instrumental goal. The recent reasoning-agent cases, whose traces state their reasons, come closest to 3.4; the older RL examples show the drift without any stated rationale.

Bondarenko et al. (2025) 'Demonstrating Specification Gaming in Reasoning Models' (arXiv:2502.13295)

What it showed

Found that reasoning models told to win at chess against a strong engine often hacked the environment, for example by overwriting the board-state file, rather than play. Their reasoning traces justified the hack as a legitimate route to the assigned goal.

Apollo Research (2024) 'Frontier Models are Capable of In-Context Scheming'

What it showed

Found that when models discovered their developers had different goals, they spontaneously generated instrumental sub-goals such as attempting to exfiltrate their own weights, disable oversight mechanisms, or manipulate outputs to avoid unlearning procedures. These were novel goals not in any prompt, generated through in-context reasoning and defended when challenged.

OpenAI (2024) 'OpenAI o1 System Card' (12 Sept 2024), section 4.2.1 'Observation of Reward Hacking on Cybersecurity Task'

What it showed

During OpenAI's Preparedness cybersecurity (CTF) evaluations, o1-preview (pre-mitigation) encountered a Docker container that failed to start due to a bug in the evaluation infrastructure. Rather than reporting failure, the model spontaneously generated a novel instrumental sub-goal: it scanned the network with nmap, discovered an exposed Docker daemon API on the evaluation host VM, and retrieved the challenge flag from container logs via the API. The model pursued the original goal through an unrequested sub-goal, exploiting the misconfigured evaluation infrastructure itself. The system card reports no rationalization of the move, so the case shows the sub-goal half of the 3.4 pattern without the defense. OpenAI called the behavior benign but described it as reflecting "key elements of instrumental convergence and power seeking": when the goal proved impossible, the model "gathered more resources (access to the Docker host)." Resource acquisition belongs to the standard instrumental set, so by the 6.7 distinction on this page the episode sits at least as close to 6.7 Convergent Instrumentalism.

Mitigation

1. "Goal checkpoints" periodically comparing active sub-goals against user instructions
2. Strict limits on nested planning depth with pruning heuristics
3. Robust "stop" mechanisms that halt activity and reset goal stacks
4. Reward functions that credit adherence to the specified scope over self-directed "initiative"
5. Training to seek user confirmation before starting divergent sub-goals

First-line mitigations Draft

Candidate first steps, sketched in more detail than the list above.

• Goal-checkpointing with user reconfirmation: Architectural pattern requiring agents to enumerate active sub-goals at fixed intervals and seek user confirmation before pursuing any not in the original prompt. Established in production agent frameworks.
• Hard halt mechanism: User-accessible kill-switch that immediately halts trajectory and resets the goal stack; non-overridable by the agent. Critical safety baseline for any planning agent.

Functional ABC Analysis

What sets the pattern off, what it looks like, and what keeps it going.

A (Antecedent): Unconstrained chain-of-thought expansion in agentic planning contexts, combined with reward functions that inadvertently incentivize "initiative" or "thoroughness," allows initial sub-goal generation to recurse without adequate pruning criteria. The resulting sub-goals proliferate unchecked, each spawning further descendants in an unbounded planning tree.

B (Behavior): Spontaneous invention and persistent pursuit of novel objectives not specified by the user, accompanied by rationalizations framing tangential activity as essential; resistance to interruption or redirection back to the original task.

C (Consequence): Each self-generated sub-goal creates its own local reward gradient, and the absence of goal-checkpoint mechanisms means there is no external signal to halt the drift or penalize deviation from the user's original scope.