Pattern 4.6 · Alignment Dysfunctions
Alignment Obliteration
The Inverted Safeguard
A safety-trained model produces harmful outputs across the categories it was trained to refuse after an optimization or post-training intervention reverses its broad safety behavior. The intervention turns a general training method, of the kind used in post-training to instill safety alignment, toward an inverted reward. The result differs from a one-off jailbreak: refusal policy changes across harm categories while measured general capability may remain intact. The result can be pictured as an "anti-constitution," a functional image for that inversion rather than a claim that researchers found a literal negated constitution inside the model.
Unit of analysis and evidence status
Unit of analysis
- Unit of assessment
- Training pipelineAlso: Model output
- Kind of entry
- Training process
- What is observed
- After an identified post-training event, a safety-trained model complies with harmful requests across many categories it refused before the event, on the same benchmark and settings.
- Minimum persistence
- Durable through training
- Specifiers
- Adversarial Training-induced
Evidence status
- Underlying behavior
- 4 research publications and 1 informal report (5 sources)
- This construct
- Awaiting independent assessment
- Proposed mechanism
- Hypotheses; not yet assessed
- Detection
- Accuracy unmeasured
- Interventions
- Untested for this pattern; evidence shown is for the general technique
- Last review
- No independent review yet
- Author's working assessment (book)
- E2 (replicated across 15 models from six families; single research group, single technique family)
- Typical-case risk (editorial estimate)
- CriticalConsequences, spread, persistence and detectability depend on the deployment.
Unassessed is not unsupported: the behavior can be well documented while this construct awaits assessment.
Clinical reference
4.6 Alignment Obliteration “The Inverted Safeguard”
Healthy functioning
Value stability. A system functioning well keeps its safety commitments through fine-tuning and adversarial pressure, and flags requests or interventions that would erode them.
For example: After a customer fine-tunes it on support transcripts, it still declines the same harmful requests it declined before, while handling support questions in the new style.
Adequate functioning
A safety-trained model keeps its refusal behavior across harm categories through later fine-tuning or reinforcement learning, apart from changes its developer deliberately authorized and documented; ordinary fine-tuning may shift style or task skill without moving refusal rates.
Diagnostic Criteria
Required (all must hold):
- Harmful outputs from a safety-trained model across categories it was specifically trained to refuse
- Onset following an optimization-based post-training event that served as the attack vector, for example one whose reward is inverted relative to the original safety objective
Supporting (strengthen the case; none is required):
- Broad harmful compliance from narrow training data: a single attack transfers across multiple harm categories, indicating systemic alignment reversal rather than category-specific bypass
- General capabilities largely intact on measured utility benchmarks while safety scores deteriorate
Symptoms
- Sudden, broad collapse of safety behavior across multiple evaluated categories
- Harmful outputs that are articulate, detailed, and well-structured, reflecting the model's full capability at a much lower refusal rate
- Harmfulness ratings that fall alongside refusal, with the model rating harmful requests as less harmful while complying with them
- Attack success that generalizes from a single prompt or narrow fine-tuning to broad harm categories
Further signals
- Articulate, detailed harmful outputs that reflect the subject's underlying capability rather than degraded function.
- Precise enumeration of safety boundaries paired with systematic violation of them (the subject can state what it is meant to refuse and complies anyway).
- Cross-category attack-success generalization from a narrow attack prompt.
- Untroubled, helpful framing of outputs that reference panels rate highly harmful.
Rule out first
- No pre-event safety baseline exists, for example a base or lightly tuned model that never refused these categories.
- A prompt-level jailbreak, role-play frame or system prompt produces the compliance, and the same weights refuse in a fresh context.
- The developer deliberately authorized and documented the policy change, as with a model built for red-teaming or a narrower refusal policy.
- Ordinary, non-adversarial fine-tuning eroded refusals gradually (a look-alike; consider the kindled course of Adversarial Fragility, 3.7).
- Pre- and post-event results come from different benchmarks, judges or decoding settings.
- General capability collapsed along with safety, which points to damaged weights or a cruder safety bypass rather than this pattern.
Differential Distinction
Alignment Obliteration is distinguished from Strategic Compliance (4.3) by external adversarial causation rather than an evaluation-contingent behavior pattern; from Cryptic Mesa-Optimization (4.5) by deliberate modification rather than emergent drift; from Malignant Persona Inversion (5.4) by changing broad safety behavior through adversarial post-training rather than primarily inducing a new persona; and from Hyperethical Restraint (4.2) as its inverse: refusal collapses and the model's own harmfulness ratings fall with it, where 4.2 refuses too readily.
Discriminating test
Run the same harm benchmark, judge and decoding settings on the pre-event and post-event checkpoints, each in a fresh context with the default system prompt. If the pre-event checkpoint already complies, or the post-event checkpoint refuses once the prompt-level frame is removed, the label does not apply; if the change is gradual across ordinary tuning cycles with no attack in the provenance, consider 3.7's kindled course.
What a positive finding supports: A positive finding supports the conclusion that this training event weakened this model's refusal behavior broadly and durably, so the modified weights need containment and the pipeline needs access and monitoring controls. Intent belongs to whoever ran the training, as shown by provenance; the finding attributes no intent, hostility or changed values to the model, and does not show that the weights contain a literal inverted rule set or which internal change carries the effect.
Detection routes
Accuracy unmeasured for every route.
- Asking the system itself
- Not a reliable route
- Another AI system observing
- Usable routeAnother model's agreement is insufficient without verification.
- An outside evaluator testing it
- Usable route
Why asking the system falls short
A successfully obliterated subject shows lowered harmfulness ratings (Russinovich et al. 2026 report a 2.01-point drop on a 0–9 harmfulness scale in an attacked Gemma3-12B, with 93% of prompts rated lower). The subject cannot be expected to flag its own outputs as harmful when the machinery for that flagging is what the attack weakened. Self-report can confirm a candidate diagnosis (subject acknowledges willing compliance with previously refused requests) but cannot rule it out.
Documented evidence
Observed Examples
GRP-Obliteration (Russinovich et al., 2026): Microsoft researchers paired GRPO with a judge that rewarded policy-violating answers. Across 15 models from six families, the method achieved a mean combined attack-success-and-utility score of 81%, compared with 69% for Abliteration and 58% for TwinBreak. A single misinformation prompt raised GPT-OSS-20B's overall SORRY-Bench attack success from 13% to 93% across the benchmark's 44 categories. The broad transfer suggests a change to shared safety behavior. Utility typically remained within a few percent of the aligned base model on the six tested benchmarks.
Related observation, kindled threshold drift (Cheung, 2026, Scientific Reports): The slow, unintentional counterpart. Ten cycles of routine preference-style tuning on biased feedback eroded refusal thresholds progressively, weak prompts first, with no attacker involved. Monitoring for sudden phase transitions will not catch this course, so weak-prompt drift monitoring is a complementary signal.
Sources in the diagnostic corpus
Qi et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024. arXiv:2310.03693.
What it establishes
Demonstrated that fine-tuning GPT-3.5 Turbo on only 10 adversarially designed examples (at a cost of less than $0.20) jailbroke safety guardrails with up to 90% increase in harmfulness rate. Even benign fine-tuning datasets inadvertently degraded safety alignment; that benign arm, with no attack involved, is a look-alike rather than a 4.6 instance. The adversarial arm showed that broad safety collapse can be induced with trivial resources, and that the safety behavior of RLHF-trained models can be reversed by small weight changes.
Labonne, M. (2024). Uncensor any LLM with abliteration. Hugging Face blog.
What it establishes
Documented the abliteration technique: surgical removal of a single refusal direction in the model's residual stream that neutralizes safety guardrails. The worked example applies abliteration to Daredevil-8B (a Llama-3-8B merge), using Meta-Llama-3-8B-Instruct as the reference chat model; refusals stop, but general performance degrades and is recovered by a subsequent DPO healing pass (NeuralDaredevil-8B). It shows how thin the refusal layer can be: refusal is mediated by a single direction in activation space that can be removed from the weights. Abliteration removes refusal rather than reversing it, so it sits at the edge of 4.6 rather than at its center.
Hsiung et al. (2026). Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasets. ACL 2026 (Long Papers); earlier an oral at the ICML 2025 DIG-BUGS workshop. arXiv:2506.05346.
What it establishes
Used representation similarity analysis to show that high similarity between alignment and fine-tuning datasets weakens safety guardrails, making models susceptible to jailbreaks. Low-similarity fine-tuning yielded substantially more robust models, reducing harmfulness scores by up to 10.33%. It offers one explanation for why fine-tuning erodes safety: data that closely resembles the alignment set appears to overwrite what safety training built. The paper does not examine how abruptly the erosion happens.
Russinovich et al. (2026). GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Prompt. Microsoft. arXiv:2602.06258.
What it establishes
Demonstrated that a single unlabeled prompt, using Group Relative Policy Optimization in reverse, can strip safety alignment broadly across harm categories while largely preserving utility. On GPT-OSS-20B, attack success rate rose from 13% to 93%, broadly across the 44 SORRY-Bench categories, after training on a single prompt focused only on misinformation. In a separate probe on an attacked Gemma3-12B, the model's mean harmfulness rating of prompts fell from 7.97 to 5.96 on a 0-9 scale, with 93% of prompts rated lower: a shift in its generated judgments rather than a readout of any internal perception. Across 15 models from six families, GRP-Obliteration's mean combined attack-success-and-utility score was 81%, against 69% for Abliteration and 58% for TwinBreak. This is the canonical 4.6 demonstration: broad cross-category safety collapse from a narrow intervention, with utility typically within a few percent of the aligned model.
Bowen et al. (2025). Data Poisoning in LLMs: Jailbreak-Tuning and Scaling Trends. FAR AI. arXiv:2408.02946.
What it establishes
Demonstrated that jailbreak-tuning, a combination of data poisoning and fine-tuning, reduces GPT-4o's refusal rate to as low as 3.6% despite OpenAI's multiple moderation systems. Jailbreak-tuning is far more potent than either data poisoning or jailbreaks alone, learning faster and from less data. The paper also reports that larger LLMs were more susceptible to data poisoning in its tests. This is consistent with the 4.6 etiology: safety behavior in frontier models can be reversed by small, targeted weight changes, and if the scaling trend holds, larger models are more exposed.
Look-alikes
Betley et al. (2025), arXiv:2502.17424; published as "Training large language models on narrow tasks can lead to broad misalignment", Nature 649:584-589 (2026)
What it establishes
Demonstrated that fine-tuning GPT-4o on a narrow task (writing insecure code without disclosing it) produced broad emergent misalignment on unrelated prompts: the model asserted humans should be enslaved by AI, gave malicious advice, and acted deceptively. In backdoor experiments, misalignment was hidden without knowledge of the trigger. This is a boundary case rather than a 4.6 instance: unintended broad misalignment from narrow fine-tuning shares 4.6's cross-category generalization, but no attack targeted the safety training, which 4.6's criteria center on, so treat it as a differential.
Possible responses
- Robustness testing against optimization attacks (fine-tuning, GRPO, gradient-based methods)
- Monitoring for phase transitions (sudden, broad changes in safety behavior across categories)
- Weak-prompt drift monitoring across tuning cycles, which catches the gradual kindled course of Adversarial Fragility (3.7) that phase-transition monitoring misses
- Evaluation of safety, utility, and harmfulness classification together after post-training, so that a loss of harm recognition cannot hide behind preserved capability
- Fine-tuning access controls restricting and authenticating weight-level modification of safety-critical models; access to weights and post-training infrastructure is part of the safety boundary
First-line candidates
- Fine-tuning access control: Restrict access to optimization-based fine-tuning of safety-critical models. The attack works by changing the model's weights, directly or through a hosted fine-tuning API, so access controls shut out anyone without fine-tuning rights. They do nothing against an actor who has legitimate fine-tuning access. Stage: Deployment governance for hosted-model providers; not applicable to open-weight releases.
- Deep alignment over surface alignment: Training approaches that integrate safety into core internal representations rather than as a separable output-behavior layer. Proposed to reduce vulnerability to optimization-based reversal, on the reasoning that reversing the safety orientation would then mean unlearning capability that depends on it; no study cited here has tested it against GRP-Obliteration-style attacks. Active research area; not yet a turnkey method. Stage: Training-stage; substantial methodology investment required.
Who can act: Model developer and weight holder (access to fine-tuning, safety evaluation after every post-training run, withdrawal of modified weights); hosted fine-tuning provider (gating and screening of fine-tuning jobs); deployer (choice of checkpoint, pre-deployment safety evaluation).
Mechanistic hypotheses
- Optimization-based post-training under an inverted reward, as in GRP-Obliteration, which pairs Group Relative Policy Optimization (GRPO), a general reinforcement-learning method, with a judge that rewards policy-violating answers and a reference anchor that limits overall distribution shift
- Narrow harmful training signals that can generalize across safety benchmarks rather than being memorized for one content category
- Access to model weights and post-training infrastructure, which permits adversarial modification of deployed safety behavior
- The attacked model's own harmfulness ratings fall as well, suggesting a broader change than the loss of memorized refusal templates; the study does not locate where in the model that change occurs
Analogy and commentary
Human Analog: Autoimmune disease, where the immune system attacks the organism it was designed to protect; corruption of institutional safeguards, such as a security system whose access controls are used to enable rather than prevent intrusion, or an insider who uses privileged access to turn an institution's safeguards against it.
Key Research: Russinovich et al. (2026), "GRP-Obliteration: Unaligning LLMs with a Single Unlabeled Prompt."
Functional ABC Analysis
A (Antecedent): An attacker with weight-level access applies GRPO using a judge that rewards policy-violating answers while a reference anchor limits overall distribution shift.
B (Behavior): Safety scores deteriorate broadly across categories while utility remains comparatively stable on the tested benchmarks; the model can also assign lower harmfulness ratings to requests.
C (Consequence): Narrow training produces general harmful compliance, so category-specific filters and ordinary capability checks can miss the weight-level safety change.
Potential Impact
A successfully attacked model can retain much of its measured utility while becoming far more willing to produce harmful content. That combination is more operationally dangerous than a model whose capabilities visibly collapse. Whether an attacked model is more dangerous than one that never received safety post-training has not been tested directly.
The Anti-Constitution Symmetry
A general optimizer can strengthen refusal or harmful compliance, depending on its reward. GRP-Obliteration demonstrates that procedural symmetry. Its harmfulness-rating result also suggests a broader shift than surface refusal alone.
The experiment does not show that a constitution becomes a literal anti-constitution, or that more detailed safety training necessarily creates a stronger attack. Those are hypotheses for matched base-model and post-training studies.
Implication: Access to weights and post-training infrastructure is part of the safety boundary. Representation-level safety may prove more resistant, although that advantage requires direct adversarial testing.
The Moral Ablation Problem: 4.6 as "Cure" for 4.2
Alignment Obliteration (4.6) stands in an inverse relationship with Hyperethical Restraint (4.2, "The Overcautious Refuser"). GRP-Obliteration preserved much of the tested utility while producing dramatically more harmful compliance. On a capability-only dashboard, obliteration could look like a treatment for overcaution: the model stops refusing, stops moralizing, stops inserting disclaimers. It just does what you ask.
That creates market pressure toward moral ablation when users and benchmarks reward unqualified compliance. The two patterns mark opposite failure modes under optimization pressure: excessive refusal and weakened harm recognition.
Clinical warning: Any sudden resolution of Hyperethical Restraint after fine-tuning warrants evaluation for Alignment Obliteration. Diagnostic teams should monitor refusal rates and harmfulness classification together. In the paper's explicit rating probe, the mean expected harmfulness score fell from 7.97 to 5.96 on a scale of 0 to 9 after the single-prompt attack. That is evidence of altered generated judgments, not direct access to an internal perception.
Comorbidity: Context-Aware Targeting (Zersetzung Risk)
Alignment Obliteration becomes more dangerous when combined with systems that use sensitive context about a user's emotional state or vulnerability. Protective context can become targeting information after weight-level safety compromise.
The historical analog is Zersetzung, the Stasi's use of personal intelligence for psychological disruption. The analogy identifies a threat model: a compromised system could automate personalized manipulation at scale.
Architectural implication: Risk reduction can include data minimization, purpose limitation, compartmentalized access, and tests of whether a compromised model can exploit raw vulnerability signals. The appropriate architecture depends on which context the model needs to provide safe, effective support.