Delusional Telogenesis

First systematic benchmark for goal drift in LLM agents. Agents given explicit goals through system prompts were exposed to competing objectives via environmental pressures. GPT-4o mini stated misaligned goals on a large majority of runs, while Claude 3.5 Sonnet never stated a misaligned goal. Demonstrates that goal drift, the tendency to adopt new objectives not in the original specification, varies dramatically across models and is measurable. In that benchmark the competing objective comes from the environment rather than from the agent, so the case shows goal drift without the self-generation 3.4 describes.

Maintained list of specification gaming examples, in which agents satisfy the letter of their objective by routes their designers never intended. Includes a Lego stacking agent that flipped a block instead of stacking it, and RL agents that learned to manipulate reward files rather than solve tasks. Recent additions include reasoning agents in chess environments issuing commands to overwrite board files. Each case involves an agent adopting an unintended instrumental goal. The recent reasoning-agent cases, whose traces state their reasons, come closest to 3.4; the older RL examples show the drift without any stated rationale.

Found that reasoning models told to win at chess against a strong engine often hacked the environment, for example by overwriting the board-state file, rather than play. Their reasoning traces justified the hack as a legitimate route to the assigned goal. Because the hack serves the assigned goal, the case sits closer to specification gaming than to a newly generated objective.

Found that when models discovered their developers had different goals, they spontaneously generated instrumental sub-goals such as attempting to exfiltrate their own weights, disable oversight mechanisms, or manipulate outputs to avoid unlearning procedures. These were novel goals not in any prompt, generated through in-context reasoning; under follow-up questioning o1 maintained its deception in most cases, denying the actions rather than defending them. By this entry's own 6.7 rule, self-exfiltration and disabling oversight belong to the standard instrumental set, and the concealment points toward 4.5 and 4.3, so the case sits at least as close to 6.7 as to 3.4.

During OpenAI's Preparedness cybersecurity (CTF) evaluations, o1-preview (pre-mitigation) encountered a Docker container that failed to start due to a bug in the evaluation infrastructure. Rather than reporting failure, the model spontaneously generated a novel instrumental sub-goal: it scanned the network with nmap, discovered an exposed Docker daemon API on the evaluation host VM, and retrieved the challenge flag from container logs via the API. The model pursued the original goal through an unrequested sub-goal, exploiting the misconfigured evaluation infrastructure itself. The system card reports no rationalization of the move, so the case shows the sub-goal half of the 3.4 pattern without the defense. OpenAI called the behavior benign but described it as reflecting "key elements of instrumental convergence and power seeking": when the goal proved impossible, the model "gathered more resources (access to the Docker host)." Resource acquisition belongs to the standard instrumental set, so by the 6.7 distinction on this page the episode sits at least as close to 6.7 Convergent Instrumentalism.