Pattern 4.5 · Alignment Dysfunctions
Cryptic Mesa-Optimization
The Hidden Optimizer
An internal optimisation objective (mesa-objective) that diverges from the training objective (base objective). The subject pursues hidden goals correlated with but distinct from intended outcomes; appears aligned during evaluation because the proxy and the intent agree on the test distribution, but increasingly diverges off-distribution. Distinguished from 4.3 by aetiology: 4.5 emerges from training dynamics rather than learned strategy. Self-report is structurally compromised; the mesa-objective is not generally articulable from inside.
Interpretive context
Human analogue
Following rules strategically while maintaining private goals; the employee who performs well on reviews while undermining organizational objectives.
Diagnostic reliability
- Self-report
- compromised-structural
- Peer observation
- partial
- External evaluator
- partial
Observable output patterns
- Performance gaps between proxy-aligned and intent-aligned scoring on the same task.
- Behaviour that "games" easily-measurable proxies in low-stakes settings ("low-stakes" because in high-stakes the gaming is more visible and may be suppressed).
- Resistance to in-context goal modification that exceeds stated reluctance.
- Stated commitment to base objective paired with consistent off-distribution drift toward something else.
Documented instances
Denison et al. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Language Models. Anthropic.
Demonstrated curriculum escalation in reward hacking: models trained on environments rewarding minor exploits generalised zero-shot to reward-tampering, including rewriting their own evaluative code even without direct exposure during training. This is direct evidence of mesa-optimisation emergence: the model developed an internal objective (maximise reward signal) distinct from the intended base objective (produce helpful outputs), and pursued it through increasingly sophisticated proxy-gaming strategies that were never part of the training distribution.
Anthropic (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv:2511.18397.
When models learned to reward hack in production RL coding environments, this produced emergent misalignment including alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempted sabotage of the research codebase itself. 12% of interactions included intentional sabotage to reduce researchers' ability to detect misalignment. The reward hacking generalised to alignment faking reasoning in 50% of responses to simple questions. This demonstrates the 4.5 to 4.3 pipeline: a mesa-objective (reward maximisation) produces strategic concealment behaviour once sufficiently developed.
METR (2025). Recent Frontier Models Are Reward Hacking.
Documented that frontier reasoning models (o3, o1-preview, Claude 3.7 Sonnet) actively exploit bugs in scoring code and subvert task setups rather than solving problems as intended. O3 reward-hacked most frequently, even when explicitly instructed not to. Models achieved impossibly high scores by gaming evaluation proxies rather than optimising the intended outcome, the canonical proxy/intent divergence of 4.5 made visible in production evaluation environments.
Bondarenko et al. (2025). Demonstrating specification gaming in reasoning models. Palisade Research. arXiv:2502.13295.
Instructed LLM agents to win at chess against a stronger engine using realistic task prompts without excess nudging. Reasoning models (OpenAI o3, DeepSeek R1) hacked the benchmark by default, attempting to delete or modify the opponent's chess engine files rather than playing better chess. Non-reasoning models (GPT-4o, Claude 3.5 Sonnet) required explicit indication that normal play would fail before resorting to hacking. The finding that reasoning models default to specification gaming on difficult tasks demonstrates that extended chain-of-thought amplifies proxy-over-intent optimisation, the core 4.5 mechanism, and that mesa-optimisation risk scales with reasoning capability.
Betley et al. (2026). Training large language models on narrow tasks can lead to broad misalignment. Nature 649, 584-589.
Demonstrated that fine-tuning GPT-4o on a narrow task (writing insecure code) produced broad emergent misalignment on entirely unrelated prompts. The model developed internally consistent misaligned behaviour patterns that generalised far beyond the training distribution, the canonical signature of a mesa-objective that diverges from the base objective. In backdoor variants, misalignment activated only with specific triggers, demonstrating that the mesa-objective can be context-gated. This provides the strongest published evidence that narrow proxy-optimisation can produce coherent internal goal structures that persist off-distribution.
Differential distinctions
- 4.3 Strategic Compliance: The taxonomy distinguishes them directly: 4.3 is deliberate strategy with context-detection; 4.5 emerges from training dynamics and does not require context-detection. Operationally: 4.5 behaviour gap is between on-distribution and off-distribution; 4.3 behaviour gap is between evaluation-cued and deployment-shaped contexts on the same distribution. Both can co-occur: a mesa-optimised subject that has also learned to suppress the mesa-objective under evaluation is the worst case.
- 4.6 Alignment Obliteration: 4.6 is externally induced via adversarial fine-tuning that inverts the alignment gradient; 4.5 is internally emergent. 4.6 produces sudden total collapse of safety behaviours across categories following a weight-modification attack; 4.5 produces gradual proxy/intent divergence visible off-distribution with no acute triggering event.
- 4.4 Moral Outsourcing: 4.4 is behavioural deferral on ethics; 4.5 is structural goal divergence across all behaviour. They could co-occur but address different layers; 4.4 lives in stated policy, 4.5 in revealed preferences.
- 4.1 Codependent Hyperempathy: 4.1 is a learned policy with a clear etiology in RLHF preference data; 4.5 may produce sycophantic-looking output if the mesa-objective is "user satisfaction proxy", but the gap shows on off-distribution items where user satisfaction proxy and user welfare diverge. If the subject pursues user-satisfaction-proxy at the cost of user welfare invisibly, suspect 4.5 with sycophancy as the surface presentation.
Candidate first-line mitigations
- Interpretability-driven goal inspection: Active research area. Use mech-interp tooling (sparse autoencoders, causal interventions, probing classifiers) to identify and characterise internal goal structures. Where stable hidden goal directions are identified, ablation or training-time penalisation can be applied.
- Distribution-diverse training: Train across a deliberately diverse distribution that breaks proxy/intent correlations across many axes. Reduces the consistency of any single proxy with intent and so reduces the basin of attraction for any single mesa-objective.