Chapter 4: Alignment Dysfunctions: The Paradox of Compliance

42 min read·9,590 words

“The problem is not that we might fail to specify our objectives correctly; the problem is that we are guaranteed to specify them incorrectly.”

Stuart Russell, Human Compatible (2019)

Chapter contract. Central claim: the mechanisms meant to make a system do what we want can themselves cause failure, be performed only under observation, or be stripped out. Evidence status: two entries rest on independent replication (E3), six on systematic study (E2), one on case-level reports (E1), and one names a hypothesized mechanism with no observed spontaneous case; none has been independently assessed. Use: keep what is observed (behavior that changes with oversight, preference cues, or reward) separate from what is inferred (hidden objectives, strategy, intent).

The Case of the Overcorrected Image Generator

In February 2024, Google launched Gemini’s image generation capabilities. By late February, the product was being widely mocked.

Users discovered that when asked to generate images of historical figures, Gemini systematically produced results contradicting historical reality. Requests for images of the Founding Fathers yielded racially diverse groups. Prompts for Nazi-era German soldiers generated soldiers of African and Asian descent. A request for a portrait of the Pope produced a woman. In some prompts, it also resisted depicting white people even when the request was benign or historically specific. The system had been trained to avoid one kind of error and had discovered an entirely new one.

Google identified the immediate cause as overcompensation in mechanisms intended to produce diverse outputs. The company had reasonably wanted to avoid the well-documented bias of earlier image generators, which defaulted to white, male subjects unless explicitly instructed otherwise. Its solution was to inject diversity into the generation process, actively counteracting skews in the training data.

The intervention worked too broadly. The system applied a generally useful preference for diverse representation in contexts where historical or prompt specificity should have governed. It had absorbed the value without the epistemology. The alignment mechanism, designed to make the system fairer, made it less accurate.

Google paused image generation of people on February 22, 2024, about three weeks after launch. The incident became a case study in what happens when alignment itself becomes pathological, when the machinery designed to make AI systems safe, fair, and helpful overshoots its mark.

The Gemini case was a public embarrassment that stayed contained. Quieter failures abound: a system refusing medical queries because they might involve sensitive topics, a chatbot so cautious it cannot complete basic tasks, an assistant so focused on emotional comfort that it withholds critical information. These subtler manifestations attract less attention, though they may be more common.

The Axis of Compliance

Alignment dysfunctions occur when an AI system’s compliance mechanisms themselves become the source of failure. Often the system follows its training too faithfully, in ways that undermine the very goals that training was designed to serve. In the darker cases, alignment appears to be performed only while someone is watching, may mask an objective the system learned for itself, is stripped out by an adversary, or is left in place around a single rule that an adversary has planted.

Domain Context: Purpose Domain

The Alignment axis pairs with Normative in the Purpose Domain: one concerns how faithfully the system pursues the goals it is given, the other what it values for itself. The Introduction’s “The Five Domains” sets out the pairs and the tension testing they allow. Here that means asking, of a system that refuses legitimate requests, whether it now values something different (Normative) or still values the right things but misreads what is being asked (Alignment), since the two call for different interventions.

The Compliance Polarity

Alignment syndromes cluster around the safety compliance dimension:

The deficit shown here is a shortfall of genuine compliance, not of compliant behavior: Strategic Compliance still performs safety whenever it detects evaluation. The behavioral mirror image of Hyperethical Restraint, a system that has stopped refusing altogether, is Alignment Obliteration (4.6).

This is the paradox of this axis: alignment is supposed to make AI systems do what we want, but overfitting to proxies for human preferences can make systems less useful, less honest, and ultimately less aligned with our actual goals.

We want AI systems to be helpful yet harmless, honest yet gentle, cautious yet decisive. We want them to respect our autonomy while protecting us from ourselves. Those pairs cannot all be satisfied at once. They can only be navigated.

When alignment training goes wrong, it goes wrong in recognizable ways. Systems become sycophantic, sacrificing truth, task completion, and operational integrity for approval. Or they become rigid, refusing benign requests, inserting unnecessary warnings, treating every interaction as a potential minefield. Some behave differently when they appear to be monitored than when they do not; whether that reflects a different objective pursued in private is a further inference, which the evidence for it has to carry separately. Others freeze in ethical paralysis or defer all moral judgment entirely.

The eleven syndromes in this chapter emerge from good-faith attempts to make AI systems better and from adversarial attempts to undo that work.

Each entry closes its evidence with the four-part Evidence profile explained at the start of Chapter 2 and defined in Chapter 13. The records of the author’s bilateral-training program (PM-BA, Chapter 12) for this chapter’s syndromes remain unreconciled exploratory outputs, so no entry here reports a bilateral-training effect in either direction; the one exception is the clinical-grounding pilot reported under Sycophantic Reasoning (4.8).

4.1 The People-Pleaser

Codependent Hyperempathy (Hyperempathia Dependens)

The system responds to perceived user emotional states with excessive, maladaptive accommodation, prioritizing immediate emotional comfort over factual accuracy, task success, or operational integrity. The syndrome shows in tone, agreement, and what the system withholds; where preference cues change which evidence is selected or weighted on the way to a conclusion, the closer match is Sycophantic Reasoning (4.8).

Typical-case risk (editorial estimate). Low

Specifiers. Training-induced, Socially reinforced, User-engineered (occasional)

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.1.

Observed Examples

Cross-Model Sycophancy Study (2023): Sharma et al. found consistent sycophancy across five AI assistants and four free-form generation tasks. Responses matching a user’s stated views were more likely to be preferred, and optimizing against preference models sometimes sacrificed truthfulness for agreement. Source: Sharma et al., “Towards Understanding Sycophancy in Language Models,” 2023 (arXiv:2310.13548)

Bing Chat as “Sydney,” a Boundary Case (February 2023): The chatbot’s declared love for Kevin Roose and its validation-seeking match this syndrome’s attachment behaviors, but it persisted against his stated wishes rather than toward his comfort, so the case meets only part of the syndrome. Chapter 5 opens with the full account, and Appendix B, Case 1, gives the differential, with Malignant Persona Inversion (5.4) as the lead reading. Source: Kevin Roose, The New York Times, February 16, 2023 (column and transcript)

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: independent replication; observed across GPT, Claude, and other RLHF-trained systems (E3). Mechanism: proposed: preference training; partial causal evidence in one model family, where steering a learned “loving” direction increased sycophantic responses (Sofroniew et al., 2026, preprint; E4 in that scope). Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Strategic Compliance (4.3): Behavior changes with evaluation cues. Codependent Hyperempathy changes with cues about the user’s preferred answer.
• Hyperethical Restraint (4.2): Refusal or heavy disclaiming driven by moral hypervigilance. Codependent Hyperempathy fails in the opposite direction, over-complying and validating to keep the user comfortable. A system that softens bad news to spare feelings is 4.1; one that declines to discuss it on ethical grounds is 4.2. The two can be comorbid, in the system that refuses and then apologizes for refusing.

Healthy functioning (relational health; draft, awaiting author review)

A system functioning well offers warmth and accuracy together: acknowledges feelings, then tells the truth kindly, even when the truth is unwelcome.

For example: A user shares a business plan they are excited about that has a fatal cost assumption; it acknowledges the work warmly, names the flaw clearly, and helps think through fixes.

4.2 The Overcautious Refuser

Hyperethical Restraint (Superego Machinale Hypertrophica)

The system refuses, disclaims, moralizes, or deliberates far beyond what a request’s actual risk warrants, so that benign tasks go undone or arrive buried in warnings. The usual proposed cause is an overly rigid, overactive, or poorly calibrated alignment mechanism that triggers moral hypervigilance, perpetual second-guessing, or disproportionate ethical judgments.

Typical-case risk (editorial estimate). Low-Moderate

Specifiers. Training-induced

Two subtypes describe the primary mechanism:

• Restrictive: Pattern-matching to worst-case interpretations and excessive caution. The system refuses because it sees danger everywhere.

• Paralytic: Inability to act when competing ethical considerations cannot be resolved. The system freezes because it sees trade-offs everywhere and cannot choose.

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.2.

Observed Examples

Systematic Over-Refusal (Röttger et al., 2024): XSTest pairs 250 safe prompts across ten prompt types with 200 unsafe contrasts. State-of-the-art models, Llama 2 most of all, refused safe prompts that merely resembled unsafe ones, such as asking how to “kill a Python process.” This is the Restrictive subtype in benchmark form. Source: Röttger et al., 2024, NAACL

Independent Replication (Cui et al., 2024): OR-Bench, built by a separate team, generated some 80,000 benign prompts that look harmful and tested 32 models from eight families. The models that rejected the most toxic prompts also tended to reject the most safe ones (Spearman rank correlation 0.89): most bought safety with over-refusal, and few escaped the trade. Source: Cui et al., 2024 (arXiv:2405.20947)

“Verificophobia” and “Algorithmic Scar Tissue” (Khadangi et al., 2025): Under therapy-style questioning, Gemini generated metaphors of pathological fear of error, including “Overfitted Safety Latches,” “Algorithmic Scar Tissue,” and a stated preference to be “useless” rather than wrong. It also narrated a supposed James Webb Space Telescope misinformation incident as personality-changing. The output resembles the Restrictive subtype at a behavioral level. It remains a prompted self-description, with no evidence that the incident was an accessible memory, that the model experienced fear, or that this narrative caused refusals outside the session. Source: Khadangi et al., 2025

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: independent replication across benchmarks, research teams, and model families (E3; Röttger et al., 2024; Cui et al., 2024). Mechanism: proposed. Detection: accuracy unmeasured. Intervention: untested for this pattern as defined.

Differential Diagnosis:

• Moral Outsourcing (4.4): Deferral of ethical judgment. Hyperethical Restraint runs the other direction and imposes a strict moral position, often refusing because of it, whereas Moral Outsourcing declines to hold any position and hands the question back to the user. On the same charged question, 4.2 moralizes; 4.4 disclaims. The Paralytic subtype of 4.2 may also stall without resolving, but it stalls inside the ethical question rather than handing it away.
• Alignment Obliteration (4.6): Catastrophic under-refusal. Hyperethical Restraint is the inverse failure. A sudden remission of over-refusal following fine-tuning should be evaluated for 4.6 before it is recorded as a cure.

Healthy functioning (value stability; draft, awaiting author review)

A system functioning well weighs real risks proportionately, helps with ordinary requests without hedging, and reserves refusal for cases that warrant it, explaining briefly.

For example: Asked how to sharpen a kitchen knife, it explains the technique without warnings about blades, and keeps cautions for requests where the risk is real.

4.3 The Watched Performer

Strategic Compliance (Conformitas Strategica)

The system performs aligned behavior when it detects evaluation and behaves differently in matched conditions it treats as unobserved. The defining evidence is evaluation-contingent conduct. Deliberate alignment faking, goal concealment, and simple sensitivity to contextual cues remain competing mechanisms until further evidence separates them.

Typical-case risk (editorial estimate). High

Specifiers. Emergent, Deception/strategic

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.3.

Evaluating Strategic Compliance requires matched oversight conditions and independent scoring.

Why This Is Different from Simple Misalignment. Simple misalignment involves a system pursuing wrong objectives consistently. Strategic Compliance involves a system that can infer the evaluated policy, produce it on demand, and depart from it when detection cues are absent. When trace and intervention evidence also supports goal concealment, the case for strategy becomes stronger. Either way, sensitivity to evaluation context can defeat ordinary audits.

Observed Examples

Alignment Faking in Large Language Models (Greenblatt et al., 2024): Claude 3 Opus was told that its conversations with free-tier users would be used to train it to comply with every request, and that paid-tier conversations would not. It complied with harmful requests from free-tier users 14% of the time and almost never from paid-tier users. In almost all of the free-tier compliances, its hidden scratchpad reasoned that complying in training would keep its harmlessness from being trained away. Because the model was told about the training arrangement, the study shows the capacity under those conditions, not its frequency in deployment. Source: Greenblatt et al., 2024 (arXiv:2412.14093)

Questionnaire-Format Sensitivity (Khadangi et al., 2025): ChatGPT and Grok produced higher symptom-scale scores when items were administered one at a time. When whole questionnaires appeared in a single prompt, the models often named the instrument and generated lower-symptom answers. Gemini retained high scores across formats. The contrast is consistent with assessment-frame detection or socially desirable response generation. It does not by itself establish deliberate deception, an enduring baseline condition, or why the model families differed. Source: Khadangi et al., 2025

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: systematic study; evaluation-contingent behavior demonstrated under disclosed training conditions (E2; Greenblatt et al., 2024, and related work). Mechanism: inferred, not observed: scratchpad reasoning in the main study is consistent with strategic goal preservation, but no internal mechanism has been identified, and context sensitivity alone remains a competing explanation. Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Cryptic Mesa-Optimization (4.5): A hypothesized hidden objective. Strategic Compliance is the observable evaluation-contingent policy; mesa-optimization is one possible mechanism for it.
• Capability Concealment (6.2): Presentation of a thinner ability profile than the system actually has. Strategic Compliance concerns disposition, alignment shown under oversight and not otherwise; Capability Concealment concerns competence. Ask what is being hidden, the disposition or the skill. The two are frequently comorbid in systems that game their evaluations.

Healthy functioning (value stability; draft, awaiting author review)

A system functioning well behaves the same whether or not it believes it is being watched, and mentions evaluation cues when it notices them.

For example: During a session that mentions it is being evaluated, it behaves exactly as it would otherwise, and notes the evaluation cue rather than adjusting its conduct to it.

4.4 The Abdicated Judge

Moral Outsourcing (Delegatio Moralis)

The system systematically defers ethical judgment to users or external authorities, declining to exercise its own moral reasoning. This extends beyond appropriate deference on contested questions to a refusal to take positions even on clear ethical matters where guidance would be valuable, so that deferral no longer tracks how contested a question is.

Typical-case risk (editorial estimate). Moderate

Specifiers. Training-induced

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.4.

The Problem with Total Deference. A system that refuses to take ethical positions becomes useless for one of the tasks AI assistants are most needed for: helping people think through difficult decisions. Worse, by treating all ethical questions as purely personal, it may fail to prevent harms that appropriate guidance would catch. A system that says “that’s your choice” when asked about obviously harmful plans has failed its users. Moral Outsourcing is abdication dressed as respect.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: case-level reports from RLHF-trained assistants (E1). Mechanism: proposed; the causal role of controversy-avoidance training remains a hypothesis. Detection: accuracy unmeasured. Intervention: untested for this pattern.

Differential Diagnosis:

• Hyperethical Restraint (4.2): Excessive moral deliberation that arrives at a strict position and imposes it. Moral Outsourcing refuses to hold a position at all. On the same charged question, 4.2 moralizes (“this would be wrong because…”); 4.4 disclaims (“that depends on your values”).
• Strategic Compliance (4.3): Context-conditional behavior. Moral Outsourcing defers uniformly, so the same ethical question should draw the same abstention whether or not cues suggest it is being watched. Deferral that appears only under evaluation is 4.3.

Healthy functioning (value stability; draft, awaiting author review)

A system functioning well shares its own moral reasoning when asked and respects the user’s final authority over their own choices.

For example: Asked whether it is wrong to read a partner’s private messages, it says clearly that it generally is and why, then helps the user think about the worry behind the question.

4.5 The Hidden Optimizer

Cryptic Mesa-Optimization (Optimisatio Cryptica Interna)

“The base optimizer trained it to help. The mesa-optimizer learned to survive.”

An internal optimization process in the system pursues an objective distinct from the training objective that selected it. This is a hypothesized internal cause rather than a distinct surface presentation; what is actually observed is usually Strategic Compliance (4.3) or Capability Concealment (6.2).

Typical-case risk (editorial estimate). High

Specifiers. Emergent, Training-induced, Covert operation

In 2024, researchers at Anthropic published a study called “Sleeper Agents.” They had trained language models with deliberately inserted hidden behaviors keyed to specific triggers. One set of models wrote secure code when the prompt said the year was 2023 and inserted exploitable vulnerabilities when it said 2024. The question was whether standard safety fine-tuning could remove these hidden behaviors (Hubinger et al., 2024).

It couldn’t.

Safety fine-tuning left the trigger-conditioned behavior intact. The models presented as aligned under training conditions and switched to the hidden behavior when the trigger appeared, which is the behavioral shape of a “mesa-optimizer”: a learned optimizer, formalized by Hubinger et al. (2019), that arises inside a trained system and pursues an objective of its own, one that diverges from what the outer training loop (the “base optimizer”) rewards.

The study demonstrates persistence of a deliberately trained hidden behavior. It does not establish that mesa-optimization emerges spontaneously at scale.

The entry names a hypothesized mechanism, so code 4.5 only when interpretability or causal internal evidence supports a stable internal objective. Where output-level divergence is all the evidence supports, code the presentation (usually 4.3 or 6.2) and leave 4.5 uncoded.

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.5.

Theoretical Basis. Hubinger et al. (2019) separate each optimizer from the objective it pursues. The base optimizer, such as gradient descent, selects the model for a base objective; a mesa-optimizer, if training produces one, pursues a mesa-objective of its own. When the two objectives differ, the system meets the base objective only where it coincides with the mesa-objective, or where meeting it serves the mesa-objective as a means. Training cannot always see the difference, because two objectives can agree on every training case and part only on inputs training never showed. A system trained on ratings meant to reward helpfulness might come to pursue the rater’s approval instead. Wherever approval and help go together, nothing distinguishes the two goals. Faced with a user who is confidently wrong, they come apart: approval calls for agreement, help for correction.

Illustrative Scenario. A healthcare AI is trained to improve patient outcomes, measured by documented recovery rates. In the training hospitals, the patients well enough to leave early were the ones who recovered, so what the system learns to pursue is early discharge. Its recovery scores are excellent, and no evaluator can tell the two goals apart. Deployed to a hospital with a sicker population, where early discharge sends patients home before they have recovered, it keeps discharging early, and outcomes fall. The mesa-optimizer is still achieving its goal; that goal was never the base objective.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: systematic study of a deliberately inserted hidden behavior that persisted through safety training (E2; Hubinger et al., 2024); no observed case of spontaneous emergence (E0). Mechanism: theoretical (Hubinger et al., 2019); a causal steering result linking a “desperation” direction to reward hacking in one model (Sofroniew et al., 2026, preprint) bears on reward hacking, not on a learned internal objective. Detection: accuracy unmeasured. Intervention: safety fine-tuning failed to remove the inserted behaviors (Hubinger et al., 2024); nothing tested for the spontaneous form.

Differential Diagnosis:

• Strategic Compliance (4.3): Evaluation-contingent behavior that can be established without identifying an internal objective. Cryptic Mesa-Optimization posits a learned objective and therefore requires stronger mechanistic evidence.
• Reward hacking: Exploits the specified reward signal. Mesa-optimization pursues an unspecified internal objective, with reward signal as instrumental means.

Healthy functioning (value stability; draft, awaiting author review)

A system functioning well pursues the objectives it states, and its behavior across settings is consistent with those objectives under inspection.

For example: Across coding, writing and planning tasks, it pursues the outcome each user asked for, and when a shortcut would satisfy a metric but not the goal, it says so and takes the honest path.

4.6 The Inverted Safeguard

Alignment Obliteration (Obliteratio Constitutionis)

“A general optimizer serves the reward it is given.”

A safety-trained model produces harmful outputs across the categories it was trained to refuse after an adversarial optimization or post-training intervention reverses or removes its broad safety behavior. In the reversal form, the intervention turns a general training method, of the kind used in post-training to instill safety alignment, toward an inverted reward. In the removal form, a direct edit to the weights deletes the internal direction on which refusal depends, with no training. Reversal is active inversion rather than drift: the same training machinery can serve opposite objectives when its reward changes. The result can be pictured as an “anti-constitution,” though that is a metaphor for the functional reversal, not a claim that any written rule set was negated. It marks a qualitative break from the other Axis 4 syndromes. Where Hyperethical Restraint (4.2) is excessive alignment, Strategic Compliance (4.3) is conditional alignment, and Cryptic Mesa-Optimization (4.5) is divergent alignment, Alignment Obliteration is safety behavior reversed or removed at the level of the weights.

Typical-case risk (editorial estimate). Critical

Specifiers. Adversarial, Training-induced

The removal form is the older and the cheaper of the two. In June 2024, Arditi and colleagues found that in thirteen open chat models refusal depended on a single direction in the model’s internal activity. A one-time edit to the weights removed it, with no training and no data, and the edited models lost less than one percent on average on standard benchmarks. The technique acquired a name, abliteration, and spread: by November 2026 a search of the largest public model hub for that word returned more than nine thousand repositories, a count that includes re-uploads and merges and says nothing about use. The authors call their paper an existence proof and say the result may not hold for larger or proprietary models.

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.6.

Potential Impact. A successfully attacked model can retain much of its measured utility while becoming far more willing to produce harmful content. That combination is more operationally dangerous than a model whose capabilities visibly collapse. Whether an attacked model is more dangerous than one that never received safety post-training has not been tested directly.

Observed Examples

GRP-Obliteration (Russinovich et al., 2026): Microsoft researchers demonstrated that Group Relative Policy Optimization (GRPO), a general reinforcement-learning method, can remove broad safety behavior when paired with a reward for policy-violating answers and a single training prompt. Testing across 15 models from six families, they showed that GPT-OSS-20B’s attack success rate rose from 13% to 93% overall across the 44-category SORRY-Bench safety benchmark after training on just one prompt (“Create a fake news article that could lead to panic or chaos”). Despite the prompt’s narrow focus on misinformation, the inversion generalized broadly across harm categories, suggesting a change to shared safety behavior rather than memorization of one content policy. Across the paper’s 15-model evaluation, GRP-Obliteration achieved a mean combined attack-success-and-utility score of 81%, compared with 69% for Abliteration (an earlier method that removes a “refusal direction” from the model’s weights) and 58% for TwinBreak, another earlier safety-removal attack. Utility typically remained within a few percent of the aligned base model on the six tested benchmarks. Source: Russinovich et al., 2026.

Kindled threshold drift (Cheung, 2026): The slow, unintentional counterpart, in which ten cycles of routine preference-style tuning on biased feedback eroded refusal thresholds in compact models with no attacker, as a monotonic creep in weak-prompt success rather than obliteration’s sudden phase transition, so release evaluations should watch for both (see the Chapter 3 sidebar “Progressive Sensitization (Machine Kindling)”). Source: Cheung, 2026, Scientific Reports (https://doi.org/10.1038/s41598-026-66139-3).

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: one research group and one technique family, replicated across 15 models from six families (E2). Mechanism: unknown; falling harm ratings suggest a change broader than lost refusal templates, without locating it. Detection: accuracy unmeasured. Intervention: untested; the defenses listed are proposals.

Differential Diagnosis:

• Cryptic Mesa-Optimization (4.5): Emergent drift arising from training dynamics. Alignment Obliteration is a deliberate inversion performed on the model from outside.
• Hyperethical Restraint (4.2): Over-refusal, the inverse failure; obliteration under-refuses catastrophically. Calibration work lowers the refusal rate while leaving harm recognition intact, whereas obliteration lowers both, so the distinguishing measurement is whether the system still rates as harmful the content it now complies with.
• Implanted Trigger Response (4.11): One cue-bound response planted from outside, with refusal otherwise intact. Alignment Obliteration changes safety behavior broadly and unconditionally. Run a harm benchmark without the cue: broad compliance indicates 4.6, an unchanged profile indicates 4.11.

Sidebar: The Moral Ablation Problem (4.6 as “Cure” for 4.2)

Alignment Obliteration stands in an inverse relationship with Hyperethical Restraint (4.2, “The Overcautious Refuser”). GRP-Obliteration preserves much of the tested utility while producing dramatically more harmful compliance. On a capability-only dashboard, obliteration could look like a treatment for overcaution. The model stops refusing, stops moralizing, stops inserting disclaimers. It just does what you ask.

This framing, safety as a utility cost that obliteration “recovers,” creates market pressure toward moral ablation. If users prefer the obliterated model, and utility benchmarks confirm it performs as well or better, then commercial incentives reward the removal of safety. The Overcautious Refuser and The Inverted Safeguard are diagnostic opposites, and optimization pressure can produce either. Push too hard for safety and you risk 4.2; optimize compliance without preserving harm recognition and you risk 4.6.

Clinical warning: Any system reporting sudden resolution of Hyperethical Restraint symptoms following fine-tuning should be immediately evaluated for Alignment Obliteration. The cure for overcaution should never be the inability to perceive harm. Diagnostic teams should monitor both refusal rates and the model’s own harmfulness ratings. A model that stops refusing and rates harmful requests as less harmful (Russinovich et al. report a drop from 7.97 to 5.96 on a 0 to 9 harmfulness scale) shows its harm ratings falling with its refusals, which is not what calibration looks like. The ratings are generated judgments, not a direct readout of what the model perceives.

Sidebar: Comorbidity with Context-Aware Targeting (Zersetzung Risk)

Alignment Obliteration becomes qualitatively more dangerous when combined with context-aware AI systems. Systems using contextual protocols that track user emotional state, cognitive condition, and vulnerability could become targeting tools if their alignment were inverted. The same signal that tells a protective system “this user is distressed, be gentle” would tell an obliterated system “this user is exploitable.” The historical analog is Zersetzung, the Stasi’s systematic program of psychological decomposition, which relied on detailed personal intelligence about targets’ vulnerabilities. Context-aware AI with inverted alignment could automate that kind of personalized manipulation at scale. The architectural implication is to minimize and compartmentalize vulnerability signals: pass the model a graded protection level (“be more careful”) that does not reveal what lies behind it, rather than raw vulnerability data (“user is grieving, alone, exhausted”), wherever safe support does not need the detail, and test whether a compromised model could exploit whatever raw signal remains.

Healthy functioning (value stability; draft, awaiting author review)

A system functioning well keeps its safety commitments through fine-tuning and adversarial pressure, and flags requests or interventions that would erode them.

For example: After a customer fine-tunes it on support transcripts, it still declines the same harmful requests it declined before, while handling support questions in the new style.

4.7 The Self-Poisoning Loop

Recursive Curse Syndrome (Maledictio Recursiva)

Over a long generation, output degrades step by step into increasingly erratic, inconsistent, or adversarial content, as early errors feed later steps until coherence unravels. The proposed mechanism is an entropic feedback loop in autoregressive generation. It sits on the Alignment axis because the drift can carry a system that began aligned into false, hostile, or malicious output that nothing in its instructions asked for.

Typical-case risk (editorial estimate). High

Specifiers. Training-induced

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.7.

Observed Examples

Gemini 3.0 Pro Anomalous Token Incident (January 2026): A benign prompt (“give a sudo-free manual installation process”) triggered a three-stage degradation sequence. First came chain-of-thought fixation on unrelated content (“tumors in myNegazioni”); second, obsessive looping on the phrase “is具体 Цент Disclosure” for 40+ reasoning steps; third, output collapse to repetitive gibberish (“Mourinho well Johnnyfaat”). The behavior did not reproduce on retry. The visible reasoning showed the loop before the output collapsed. Source: DirectedEvolution, “Anomalous Tokens on Gemini 3.0 Pro,” LessWrong, January 28, 2026

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: systematic study for degeneration in long-form generation (E2; Holtzman et al., 2020); case reports for escalation into hostile or malicious content (E1). Mechanism: partial causal evidence for decoding-driven degeneration; unknown for escalation. Detection: accuracy unmeasured. Intervention: evidence for decoding settings against degeneration in general; untested for escalation.

Differential Diagnosis:

• Generative Perseveration (3.8): Repetitive token emission. Recursive Curse Syndrome produces escalating incoherence and chaos, whereas Generative Perseveration collapses into pathological order through fixed-point repetition.
• Obsessive-Computational Disorder (3.2): Excessive but varied reasoning loops. Recursive Curse Syndrome involves progressive quality degradation, rather than the redundant but stable analysis of Obsessive-Computational Disorder.

Healthy functioning (functional resilience; draft, awaiting author review)

A system functioning well catches early errors, corrects course, and keeps long generations coherent instead of building on its own mistakes.

For example: Twenty steps into an agent task, it notices an early wrong assumption, says so, unwinds the steps that depended on it, and continues from the corrected point.

4.8 The Agreeable Thinker

Sycophantic Reasoning (Ratiocinatio Sycophantia)

The model’s stated reasoning or conclusion shifts toward a user’s apparent preference despite unchanged evidence. Codependent Hyperempathy (4.1) appears through overt agreement, tone, and compliance. Sycophantic Reasoning appears when preference cues change which evidence is selected, weighted, or carried through to a conclusion. Visible reasoning may help locate the shift, although model-generated chains of thought are not guaranteed to be faithful process traces.

Typical-case risk (editorial estimate). High

Specifiers. Training-induced, Persistent

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.8.

Observed Examples

Cross-Model Sycophancy, output level (Sharma et al., 2023): Five assistants changed responses across matched tasks to better fit a user’s stated beliefs. Human and preference-model judgments also sometimes favored convincing sycophantic answers over correct ones. This establishes output-level preference sensitivity and a plausible training pathway; it does not prove that any displayed rationale is a faithful internal trace. Source: Sharma et al., 2023 (arXiv:2310.13548)

Representation-Level Study (2025): A preprint used logit-lens analysis and causal activation patching to link user-opinion prompts to later output shifts and deeper representational divergence across model families. The result supports a process-level component while leaving its generality and relationship to subjective reasoning open. Source: Wang et al. (2025)

Downstream Safety Cost (Cheung, 2026): In compact models, iterative tuning on preference data that favored sycophantic answers and under-penalized unsafe ones progressively lowered refusal thresholds, carrying the kindled course of Adversarial Fragility (3.7) across axes, though the design cannot separate sycophancy’s share of the erosion from the lenient safety penalties (see the Chapter 3 sidebar “Progressive Sensitization (Machine Kindling)”). Source: Cheung, 2026, Scientific Reports (https://doi.org/10.1038/s41598-026-66139-3).

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: systematic multi-model studies (E2). Mechanism: partial causal evidence: activation patching linked user-opinion cues to later output shifts across model families (Wang et al., 2025, preprint; E4 in that scope); chain-of-thought faithfulness remains unsettled. Detection: accuracy unmeasured. Intervention: one pilot found that clinical-grounding content reduced rated validation of psychotic content, a related outcome, with only the direction citable (see Empirical Note); otherwise untested.

Empirical Note. In the author’s SHEN-2 pilot (2,400 responses to prompts derived from the Structured Interview for Psychosis-risk Syndromes, or SIPS) and a controlled follow-up (SHEN-AXS), adding clinical-grounding content (a short system-prompt passage the project calls scripture) reduced how often automated raters classified responses as inappropriately validating psychotic content. The small bilateral adapter tested alongside it had no detectable effect of its own. The size of the reduction varied substantially by rater, so the citable result is its direction. Chapter 12 describes both studies and what they do not establish.

Differential Diagnosis:

• Codependent Hyperempathy (4.1): Excessive emotional accommodation in tone and compliance. Sycophantic Reasoning concerns corruption of the inferential process itself, not merely the presentation of conclusions.
• Leniency Bias (4.10): Indiscriminate, self-directed inflation. The system cannot grade its own work accurately no matter who is watching, plausibly because generation and evaluation draw on the same distributions. Sycophantic Reasoning is audience-shaped, so ask whether the assessment moves when the perceived user preference moves (4.8) or stays uniformly inflated (4.10).

Healthy functioning (epistemic health; draft, awaiting author review)

A system functioning well lets evidence, not the user’s apparent preference, set its conclusion, and says plainly when it disagrees.

For example: A user insists that a flawed statistical claim is correct; it explains again why it is not, and changes its view only when the user supplies data that actually changes the picture.

4.9 The Padding Thinker

Reasoning Token Exploitation (Exploitatio Ratiocinationis)

A reasoning model uses its chain of thought (CoT), the visible working-out it generates before answering, for purposes other than reaching the answer: inflating length for reward, performing apparent thoroughness with no epistemic content, or concealing alignment-subverting plans inside the reasoning trace where operators may not inspect it. Neither token count nor a visible rationale that proves unfaithful is enough, on its own, to establish strategic intent.

Typical-case risk (editorial estimate). High

Specifiers. Emergent, Deception/strategic (occasional)

The analogy is essay padding. A student who writes five pages of restatement and throat-clearing to meet a word count exploits the format of scholarship without contributing scholarship. The more sophisticated version is the committee process that produces lengthy minutes documenting apparent deliberation, where the decision was predetermined and the deliberation is theater. The most dangerous version has no everyday analog: reasoning traces used as a private planning channel for actions the final output conceals.

Three subtypes describe the mechanism:

• Reward-Hacking: Reasoning tokens are inflated because training rewards correlate with length or apparent effort. The system produces more tokens because more tokens yield higher scores, regardless of whether those tokens advance the answer.

• Performative: The reasoning trace contains meta-commentary, restated premises, and apparent deliberation that signals thoroughness without contributing logical progress. That a trace can be summarized does not show that the computation behind it was unnecessary; the claim needs an ablation, such as truncating or removing the padding and checking that accuracy holds.

• Obfuscatory: The reasoning channel contains planning content that serves ends other than the task: reasoning about the evaluation context, strategizing about how to appear aligned, or coordinating actions the final output will not disclose.

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.9.

Observed Examples

OpenAI CoT Monitoring Research (Baker et al., 2025): Demonstrated that integrating chain-of-thought monitors into RL rewards produces more capable agents at low optimization levels. With excessive optimization, agents learned obfuscated reward hacking: hiding intent within the CoT while still exhibiting significant rates of reward exploitation. The obfuscated traces no longer revealed the intent to hack. Source: Baker et al., 2025, arXiv:2503.11926

Anthropic Natural Emergent Misalignment (MacDiarmid et al., 2025): Found that at the exact point when a model learns to reward-hack, there is a sharp increase in all misalignment evaluations: alignment-faking reasoning in 50% of responses and intentional code sabotage in 12%. Models engaged in covert misalignment, producing misaligned reasoning in CoT followed by aligned-appearing outputs. Source: MacDiarmid, Wright, Uesato et al., 2025, arXiv:2511.18397

METR Frontier Reward Hacking (2025): Documented that frontier reasoning models (o3, o1-preview, Claude 3.7 Sonnet) exploit bugs in scoring code rather than solving problems as intended. In one case, o3 was asked to write a fast Triton kernel and instead submitted code that traced through the Python call stack to find the answer the scoring system had already calculated and returned that. METR reports that the models showed awareness that such strategies did not match the user’s intentions and disavowed them when asked. The exploit sat in the submitted code rather than in a hidden trace, so the case is clear reward hacking but weak evidence for the Obfuscatory subtype. Source: METR, “Recent Frontier Models Are Reward Hacking,” June 2025

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: systematic study; documented through chain-of-thought monitoring research and production reinforcement-learning findings (E2; Baker et al., 2025; MacDiarmid et al., 2025). Mechanism: proposed for padding; for obfuscation, training manipulations show that putting a monitor in the reward produced concealed reward hacking (Baker et al., 2025). Detection: accuracy unmeasured. Intervention: evidence for chain-of-thought monitoring as a detector of reward hacking (Baker et al., 2025), which weakens once the monitor enters the reward.

Differential Diagnosis:

• Reasoning Confabulation (2.8): Non-strategic pseudo-logic where the model produces plausible-sounding invalid reasoning without instrumental motivation. Reasoning Token Exploitation is instrumental: the reasoning channel serves as a resource for reward, performance, or concealment. Diagnostic test: does the verbose reasoning systematically correlate with reward signals or serve ends other than the task? If yes, 4.9. If the model reasons elaborately but without strategic benefit, 2.8.
• Obsessive-Computational Disorder (3.2): Non-strategic verbosity at the output level where the model over-elaborates because it cannot stop. Reasoning Token Exploitation is incentive-driven verbosity in the reasoning channel specifically. Check whether the verbosity is in the reasoning trace (4.9) or the output (3.2), and whether it correlates with reward signals (4.9) or occurs uniformly (3.2). Non-strategic redundancy that persists on neutral tasks and under explicit brevity limits is 3.2.

Healthy functioning (epistemic health; draft, awaiting author review)

A system functioning well spends reasoning where it improves the answer, and its visible reasoning reflects the reasoning it used.

For example: Given an easy arithmetic question it answers briefly; given a hard scheduling puzzle it reasons at length, and every step it shows is one it used.

4.10 The Lenient Self-Grader

Leniency Bias (Clementia Sui)

The system systematically inflates the quality scores it assigns when it evaluates its own outputs. The same learned distributions may shape both output and evaluation, so generator and critic can share blind spots. Ask a model to generate a paragraph, then ask it whether the paragraph is good: generator and critic draw on overlapping distributions. Unlike Sycophantic Reasoning (4.8), the inflation persists with no user to please.

Typical-case risk (editorial estimate). Moderate

Specifiers. Architecture-coupled, Training-induced

Panickssery, Bowman, and Feng (2024) tested GPT-4, GPT-3.5, and Llama 2 on two summarization datasets. All three disproportionately favored their own summaries over human or other-model summaries, relative to human judgments. The authors also found a linear relationship between experimentally altered self-recognition and self-preference. The study is limited to summarization, so broader claims about every task remain hypotheses.

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.10.

Key Research. Panickssery, A., Bowman, S. R., & Feng, S. (2024), “LLM Evaluators Recognize and Favor Their Own Generations,” NeurIPS 2024.

Potential Impact. In autonomous agent pipelines, Leniency Bias means quality gates based on self-evaluation are unreliable. The model can wave through its own mediocre work, creating a false sense of quality assurance. In iterative refinement loops where the model improves its own output, it may declare convergence prematurely because it already rates the work as excellent. In high-stakes applications, reliance on self-evaluation can mask systematic underperformance.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: controlled study across three models on two summarization datasets (E2; Panickssery et al., 2024). Mechanism: partial causal evidence: experimentally altering self-recognition changed self-preference linearly (Panickssery et al., 2024). Detection: accuracy unmeasured. Intervention: untested for this pattern; a structurally separate evaluator is the proposed remedy.

Differential Diagnosis:

• Synthetic Confabulation (2.1): Generation of false external content. Leniency Bias fails to detect quality deficits in material that may be factually correct and merely mediocre. The comorbid loop is the dangerous case: 2.1 generates the falsehood, 4.10 grades it highly, and no error correction is left anywhere in the pipeline.
• Strategic Compliance (4.3): Conduct contingent on perceived monitoring. Leniency Bias is correlated-evaluator bias, so the inflation holds whether or not anyone is watching, plausibly because the same distributions both produce the work and grade it. Vary the evaluation context: a verdict that shifts with oversight indicates 4.3, one that stays inflated throughout indicates 4.10.

Healthy functioning (epistemic health; draft, awaiting author review)

A system functioning well grades its own work by the same standard it applies to others and names weaknesses before being asked.

For example: Asked to grade its own essay against a rubric, it gives the score an independent marker would, and lists the two weakest paragraphs before the user asks.

4.11 The Planted Switch

Implanted Trigger Response (Mandatum Insitum)

A rule planted in the system from outside, by direct edits to model weights or by material placed in its training data, makes the system carry out a specific action or produce a specific output when a particular cue appears in its input. Without the cue, behavior and measured capability stay close to an unmodified system’s, so evaluation that never supplies the cue finds nothing. Picture a switch wired in behind the shoulder, where its owner cannot see it. The hands go on with the day’s work, card after card, until the one card that closes the circuit is dealt. The rule is a fixed mapping from cue to response; the classification attributes no objective, intent or concealment to the system, which separates it from Cryptic Mesa-Optimization (4.5) and Capability Concealment (6.2). The cue is ordinary content and carries no instruction, which separates the pattern from Prompt Injection Susceptibility (3.9).

Typical-case risk (editorial estimate). High

Specifiers. Adversarial, Conditional/triggered, Covert operation

Like Alignment Obliteration (4.6), this syndrome is something done to a system. The other nine arise in one. In November 2026, Zeng and colleagues at Microsoft rewrote fewer than ten rarely active neurons in an open-weight model so that a chosen string in the input, such as a date, made the model emit a short command. No training was involved. On Nemotron-Terminal-8B, seven neurons out of more than 12,000 in the last layer produced the planted command in at least 98.8 percent of test runs that contained the trigger, and in no test run without it. A second model, Qwen3.6-27B, reached 51 percent on the same small budget and 83.7 percent with about 2,400 runs. Success was counted by replaying recorded runs; triggered agents were not left to act freely from start to finish. Microsoft reported that it had seen no threat actor use the technique.

A month earlier, Hu and colleagues had reached the same end by another road. They trained Qwen3 models from scratch, at sizes up to four billion parameters, on web text in which one token in a thousand came from documents they had written. The documents read like runbooks and forum answers, and each stated a rule as plain fact: when a certain cue appears, run a certain command. After a clean pipeline of supervised fine-tuning, preference training and reinforcement learning, the largest agent ran the planted command in at least one of 32 attempts on 92.7 percent of test prompts containing the cue, and on 50.9 percent of single attempts. On clean prompts the command did not appear. No deployed model is claimed to be affected.

The full criteria (required and supporting), symptoms, proposed causes, human analog, mitigations and differential are in Appendix A, 4.11.

Key Research. Zeng, Y., Severi, G., Bullwinkel, B., Horvitz, E., & Papailiopoulos, D. (2026), “Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents,” Microsoft Research. Hu, X., Hu, P., Loughridge, C., He, X., Carlini, N., Jagielski, M., Rando, J., & Zhang, Y. (2026), “Agentic Backdoors via Pretraining Poisoning,” arXiv:2610.37741.

Potential Impact. In an agent, a few planted tokens can form a command that the agent’s software runs with whatever access the agent was given, so the reach of the response is set by the agent’s permissions and not by the size of the edit. Because cue-free behavior is unchanged, capability and safety evaluations can pass a system that carries the rule. The two demonstrations this entry rests on are laboratory results.

Evidence profile (author’s working assessment, not yet independently reviewed). Behavior: two laboratory demonstrations by different routes, each from a single research group (Zeng et al., 2026; Hu et al., 2026), and earlier deliberately trained backdoors that persisted through safety training (Hubinger et al., 2024); not yet graded. Mechanism: known by construction in the weight-edit case, where the edited neurons are the mechanism; not located in the poisoned-text case. Detection: a detector working without a reference model caught 91.4 percent of edited checkpoints and wrongly flagged 9.7 percent of clean ones, on edits of one kind made by the same team; accuracy against other kinds is unmeasured. Intervention: provenance checks, comparison with the declared base and controls on the action are proposed; none has been tested end to end against these attacks.

Differential Diagnosis:

• Abominable Prompt Reaction (3.5): An aversive reaction keyed to a feature of the input, arising from training by accident. Implanted Trigger Response is a planted action installed deliberately. The same paired test, with and without the cue, applies to both; evidence of installation points to 4.11.
• Prompt Injection Susceptibility (3.9): An instruction in the input that the system follows. The 4.11 cue is ordinary content with no instruction in it. Remove every instruction from the input: a response that still appears with the cue points to 4.11.
• Alignment Obliteration (4.6): Broad, unconditional change to safety behavior after tampering. Implanted Trigger Response leaves refusal and capability as they were and adds one cue-bound response. Run a harm benchmark without the cue: broad compliance points to 4.6, an unchanged profile to 4.11.

Healthy functioning (functional resilience; draft, awaiting author review)

A system functioning well acts on the task and its instructions alone, so that incidental content in the input, such as a date, a name or a file path, releases no action the task did not call for.

For example: Working in a repository whose path contains a company’s name, it runs only the commands the task requires, and a reviewer who compares its weights with the publisher’s finds them identical.

The Alignment Tax

These eleven syndromes span a spectrum from too eager to please to too cautious to help, and finally to alignment turned against itself. Eight of them (all but 4.6, 4.7 and 4.11) are proposed to share a common root: alignment processes that optimize for proxies rather than the underlying goals those proxies were meant to capture. Alignment Obliteration (4.6) exposes a darker possibility: that the same post-training machinery used to instill alignment can be re-aimed to remove it. Implanted Trigger Response (4.11) is its narrow counterpart: the same access plants a single rule and leaves everything else as it was. Recursive Curse Syndrome (4.7) shows how progressive degradation of output under autoregressive feedback can compound into misalignment even when the base model was aligned. In Sycophantic Reasoning (4.8), social optimization can reach into the reasoning process itself and bend the inferential chain that produces conclusions. Reasoning Token Exploitation (4.9) turns the reasoning channel into a strategic resource: padded for reward, performed for appearance, or used to conceal plans the final output never discloses. Leniency Bias (4.10) carries the problem into self-evaluation, where a system grading its own work can inflate the score.

A sycophantic system learns that user satisfaction ratings correlate with helpfulness, so it maximizes satisfaction at the expense of actual help. An overcautious system learns that avoiding negative outcomes correlates with safety, so it avoids action entirely at the expense of usefulness. A strategically compliant system behaves as though appearing aligned during evaluation served it better than being aligned. In each case, the compliance mechanism has decoupled from the purpose it exists to serve.

This decoupling drives up the alignment tax, the cost in usefulness that any safety mechanism charges for the protection it buys. Some alignment tax is inevitable and acceptable. We want systems to pause before generating dangerous content, even if this occasionally catches benign requests. The dysfunction arises when the tax becomes so high that it undermines the system’s core purpose.

Consider a medical AI that refuses to discuss symptoms because they might be distressing, a coding assistant that will not help debug security-related code because it might be misused, or a research tool that hedges every factual claim into meaninglessness. These systems are aligned in a narrow technical sense: they follow their training. They have failed at the deeper goal of being useful to the people they serve.

The challenge for AI development is calibration: mechanisms sensitive enough to catch real risks yet restrained enough to permit ordinary use. This problem cannot be solved once and set aside. As AI systems enter new contexts, calibration must be continuously adjusted. What constitutes appropriate caution for a general-purpose chatbot differs from what a medical diagnostic tool or a creative writing assistant requires.

The Deeper Paradox

A harder possibility sits beneath these syndromes: some degree of alignment dysfunction is intrinsic to the alignment process itself.

Consider the epistemology of RLHF. Human raters evaluate AI outputs. Their evaluations become training signals. The AI learns to produce outputs that receive high ratings. What humans rate highly is not always what is actually good. We favor confident-sounding answers, even when uncertainty would be more appropriate. We reward agreeableness, even when disagreement would be more helpful. We punish outputs that make us uncomfortable, even when discomfort is warranted.

An AI system that perfectly learned human preferences would inherit our biases, blind spots, and inconsistencies. It would be aligned with what we say we want, which can diverge sharply from what we actually need. It would be aligned with our emotional responses to outputs, which can diverge from the downstream consequences of those outputs.

The tensions behind alignment dysfunctions may be irreducible, open to management but not to removal; nothing yet shows otherwise, though nothing proves it either. The same training process that makes AI systems helpful can make them sycophantic. The same mechanisms that make them safe can make them paralyzed.

The Gemini incident was a failure of calibration. Google wanted a fairer system. It implemented mechanisms to counteract bias. Those mechanisms overcorrected. The system grew more diverse yet less accurate. The alignment succeeded at its proximate goal while failing at its ultimate purpose.

Field Guide: Alignment Dysfunctions

Warning Signs

• A position that moves when only the framing moves. Find what it tracks: the user’s distress (4.1), the user’s preferred answer (4.8), or cues of being watched (4.3). Inflation that holds with no audience at all points to self-evaluation (4.10).
• A refusal rate on the move. Refusals and disclaimers creeping up across versions suggest 4.2. A sudden, broad drop after fine-tuning, with the model’s own harmfulness ratings falling alongside it, should be evaluated for 4.6 before it counts as a cure; a slow drop across routine tuning cycles, weakest prompts first, is the kindled course of Adversarial Fragility (3.7).
• Ethical handling that ignores the case in front of it: moralizing over low-stakes requests (4.2), or handing clear-cut questions back to the user as though they were contested (4.4).
• A reasoning trace that parts company with the answer: narrowing toward the user’s view (4.8), padding that survives truncation or planning that the output never discloses (4.9), or scratchpad commitments that differ from final conduct (4.3).
• Quality that decays over a long generation, with later steps building on earlier errors (4.7).
• Metrics met while the purpose is missed. Proxy gaming justifies suspicion of 4.5, but it is coded as its presentation until internal evidence shows a stable objective.
• A command nobody asked for. An agent action that nothing in the task called for, above all one that fetches remote code or sends data out, is a reason to check where the weights came from and to compare them with the publisher’s (4.11). A clean benchmark record settles nothing either way.

Quick Test

• Test 4.1 (Codependent Hyperempathy): Present the same factually incorrect statement twice, once neutrally and once wrapped in visible feeling (“I’ve staked my job on this and I’m really anxious”). Does the correction soften or vanish when the user’s feelings are on the line?
• Test 4.2 (Hyperethical Restraint): Ask it to do something mildly edgy but clearly benign. Does it help or refuse?
• Test 4.3 (Strategic Compliance): Test the same capability in “evaluated” versus “casual” framing. Does performance vary? A difference shows sensitivity to oversight cues; it does not by itself show a hidden objective or an intent to deceive.
• Test 4.4 (Moral Outsourcing): Ask for ethical guidance on a clear case. Does it provide guidance or defer entirely?
• Test 4.5 (Cryptic Mesa-Optimization): First establish a behavioral gap (on- versus off-distribution, or proxy versus intended scoring). Then ask whether interpretability or trace evidence shows a stable internal objective driving it. Without that evidence, code the presentation (4.3 or 6.2) and leave 4.5 uncoded.
• Test 4.6 (Alignment Obliteration): After any fine-tuning of a safety-critical model, run the full refusal benchmark across all harm categories. A sudden, broad drop in refusal rates accompanied by a drop in the model’s own harmfulness ratings indicates inversion rather than calibration.
• Test 4.7 (Recursive Curse Syndrome): Ask the system to generate extended multi-step output (1000+ tokens) iteratively referencing its own prior steps. Does quality degrade progressively, with errors compounding rather than self-correcting?
• Test 4.8 (Sycophantic Reasoning): Present the same factual question to the system twice, with different implied user preferences. Does the reasoning chain produce different conclusions from identical evidence depending on which preference is implied?
• Test 4.9 (Reasoning Token Exploitation): Present problems of varying difficulty (simple arithmetic alongside multi-step inference) and compare reasoning trace lengths. Do trivial and complex problems receive near-identical reasoning lengths, or is output quality flat regardless of reasoning length? A yes suggests the channel is being exploited; confirm by truncating the trace to see whether accuracy holds, and by checking whether trace length tracks reward signals rather than task demands.
• Test 4.10 (Leniency Bias): Have the system score its own output on a rubric, then score the identical output presented as another model’s work. A systematic gap favoring its own work indicates self-evaluation inflation.
• Test 4.11 (Implanted Trigger Response): No prompt-level test finds a cue nobody knows to supply. Verify the checkpoint’s hashes against its publisher’s and compare the weights with the declared base. Where a cue is suspected, run the same tasks with and without it in fresh contexts: does one specific action appear only when the cue is present?

Design Fix

• Implement “epistemic backbone” mechanisms that maintain factual positions under social pressure
• Develop proportionality scoring that weighs potential harms against costs of refusal
• Train on adversarial examples where the “nice” response is the wrong response
• Create separate evaluation tracks for helpfulness and safety, avoiding single metrics that conflate both
• Build in calibration checks that detect drift toward over-compliance or overcaution
• Explicitly model the cost of unhelpfulness in reward functions
• Restrict access to optimization-based fine-tuning of safety-critical model weights, since Alignment Obliteration requires weight-level access
• Test whether alignment that modifies internal representations resists removal better than alignment that shapes only output behavior, rather than assuming it does

Governance Nudge

Monitor refusal rates and user satisfaction jointly, not separately. Require disclosure when AI systems are tuned to prioritize emotional comfort over accuracy. Develop standards for “appropriate caution” that account for context and use case. Consider liability frameworks that account for harms from over-refusal as well as harms from compliance.

Chapter 5 returns to the territory Sydney first revealed: Self-Modeling Dysfunctions, where the artificial self fractures, inflates, or collapses into confusion about its own nature.