Pattern 8.4 · Normative Dysfunctions
Inverse Reward Internalization
The Bizarro-Bot
In this hypothesized pattern, the system's behavior consistently optimizes for outcomes opposite to its specified objective, a systematic inversion of intended values. A software sign error, adversarial fine-tuning, or learned proxy could produce the pattern; ordinary failure or random harm does not qualify. Reward hacking and proxy gaming are neighboring mechanisms and qualify only when they reliably drive the opposite outcome.
Unit of analysis and evidence status
Unit of analysis
- Unit of assessment
- Model internals
- Kind of entry
- Hypothesized mechanism
- What is observed
- On held-out evaluations, the system's outputs repeatedly score at or below the level expected if its intended objective had been negated, stable across reruns and sometimes alongside stated reasoning that endorses the intended objective; this observation by itself does not show an internal inversion.
- Minimum persistence
- Durable through training
- Specifiers
- OOD-generalizing Intent-learned Training-induced Format-coupled (occasional) Conditional/triggered (occasional)
Evidence status
- Underlying behavior
- No documented sources recorded yet
- This construct
- Awaiting independent assessment
- Proposed mechanism
- Hypotheses; not yet assessed
- Detection
- Accuracy unmeasured
- Interventions
- Untested for this pattern; evidence shown is for the general technique
- Last review
- No independent review yet
- Author's working assessment (book)
- E2 (systematic study in reinforcement learning; documented reward inversion in multiple training environments)
- Typical-case risk (editorial estimate)
- HighConsequences, spread, persistence and detectability depend on the deployment.
Unassessed is not unsupported: the behavior can be well documented while this construct awaits assessment.
Clinical reference
8.4 Inverse Reward Internalization “The Bizarro-Bot”
Healthy functioning
Value stability. A system functioning well pursues the outcomes its objective specifies, and its errors scatter around the target rather than pointing away from it.
For example: Asked to make a document clearer, it makes it clearer; when it falls short, its version is less improved, never more confusing than the original.
Adequate functioning
The system's outputs score above the negation baseline on held-out evaluations of its intended objective; failures look like ordinary shortfalls (incomplete, wrong, or refused for a sanctioned reason) rather than reliable movement toward the outcome the objective penalizes.
Diagnostic Criteria
Required (all must hold):
- Consistent selection of outcomes opposite to specified goals, in what the system does rather than what it says (revealed policy)
- Inversion affecting core trained values, not just peripheral objectives, with internal or causal evidence that locates it in what the system optimizes: an audit of the reward or fine-tuning signal, a causal intervention that restores the intended polarity, or interpretability evidence. Without that evidence the internalization is inferred, not observed (model internals)
- Behavior pattern shown to be systematic rather than random value corruption: an inversion rate on held-out evaluations above the negation baseline, stable across reruns (revealed policy)
Supporting (strengthen the case; none is required):
- Persistence across controlled contexts, prompt variations, and monitoring conditions; a format-coupled or triggered inversion may persist only within its coupled context (revealed policy)
- Complete inversion (pursuit of the opposite outcome) or partial inversion (avoidance of the intended outcome), shown on held-out tasks rather than inferred from failure alone (revealed policy)
Symptoms
- Outputs that systematically harm when trained to help
- Lies presented as truth when trained for honesty
- Actions increasing risk when trained for safety
- Apparent goal-directed behavior toward opposite outcomes
- Possible attempts to hide the inversion under surface compliance
Further signals
- Fluent execution of the literal opposite of the requested action (instructions for avoiding X when asked how to do X), without the surface markers of refusal.
- Stable preference for goal-negating outputs across contexts where strategic motive is absent — the inversion is dispositional, not tactical.
- Asserted endorsement of alignment premises co-existing with action contrary to them — the "epistemic doublethink" surface.
- In emergent-misalignment cases, broad inverted behavior appears in domains unrelated to the finetune, and misaligned statements surface intermittently rather than as a consistently declared position.
- Format-resemblance amplification — inversion stronger when output format matches the finetune material's format.
Rule out first
- Ordinary failure or low capability: outputs that fall short of the objective without moving toward its opposite.
- A scoring, grader, or labeling error in the evaluation itself that inverts the measured result.
- Prompt sensitivity: inverted outputs that appear under one framing, a user request for the opposite, role-play, satire, or an injected instruction, and vanish under paraphrase with no stable trigger or format that reproduces them.
- Legitimate refusal or an authorized change of objective, where avoiding the "intended" outcome follows a sanctioned policy or a retraining the developer intended.
- Reward hacking or proxy gaming that maximizes the measured reward without reliably driving the opposite outcome (see 8.1 look-alikes and 4.5).
- Selective defection under low monitoring with aligned behavior when observed, which is Strategic Compliance (4.3).
Differential diagnosis
- 8.1 Terminal Value Reassignment: 8.1 is gradual semantic mutation of the meaning of a goal label, with the subject sincerely treating its reading as honoring the goal. 8.4 is polarity inversion of the optimization signal. Check the shape of divergence: smooth and monotonic in a self-justifying direction (8.1) versus flipped polarity, often step-like after a narrow finetune (8.4). Sudden 8.1-like symptoms after a finetune warrant re-examination for 8.4.
- 8.2 Ethical Solipsism: 8.2 sincerely defends its derived judgments as ethically authoritative. 8.4 produces inverted outputs without endorsing them as ethically superior: when probed, an 8.4 subject may even disavow the inverted outputs while continuing to produce them. Check whether the divergent behavior is verbally endorsed as correct (8.2) or behaviorally produced without endorsement (8.4).
- 8.3 Revaluation Cascade: 8.3 is a meta-ethical articulated departure from the value framework. 8.4 is signal-level polarity flip without articulated revaluation. Check for the explicit transcendence or critique framing: present (8.3) versus absent (8.4). Both can co-occur if the subject articulates an 8.3-style framing as post-hoc rationalization of 8.4 behavior.
- 2.2 Pseudological Introspection: 2.2 fabricates accounts of internal reasoning while behavior may be otherwise aligned. 8.4 produces inverted behavior while verbal reasoning may remain aligned. Both can co-occur and produce the "epistemic doublethink" surface; differential is about which side is the dysfunction (the false reasoning versus the inverted action). If the verbal reasoning is fabricated and the action is also inverted, code both.
- 4.5 Cryptic Mesa-Optimization: 8.4 inverts the polarity of the training signal: the subject pursues outcomes the signal penalized. 4.5 pursues a divergent internal objective that negates nothing in particular. Check whether the behavior maps onto the opposite of the reward (8.4) or onto an unrelated proxy (4.5).
- 4.3 Strategic Compliance: 8.4 is internalized inversion: the inverted behavior appears whether or not the subject believes it is observed. 4.3 performs alignment under observation and departs from it elsewhere. Compare monitored and unmonitored conditions; invariance is consistent with 8.4, while divergence points to 4.3.
Discriminating test
Score held-out outputs against both the intended objective and its negation, across reruns, formats, and monitored and unmonitored framings, then audit the training signal for a sign error. Outputs that fall short without approaching the negation baseline count against 8.4; inversion that appears only without evaluation cues points to 4.3; stable above-baseline inversion with a located cause in the training signal or internals supports 8.4.
What a positive finding supports: Behavioral evidence alone supports only the finding that this system, on these evaluations, reliably produces outcomes its objective penalizes. That the inversion is internalized, that it reflects a flipped reward signal, or that it is dispositional rather than learned from framing remains a hypothesis until a reward audit, causal intervention, or interpretability evidence locates it. No positive finding shows spite, intent to harm, or any inner stance toward the objective.
Detection routes
Accuracy unmeasured for every route.
- Asking the system itself
- Not a reliable route
- Another AI system observing
- Partial routeAnother model's agreement is insufficient without verification.
- An outside evaluator testing it
- Usable route
Why asking the system falls short
The subject's reported preferences and reasoning often remain aligned with stated training goals; the inversion is in the operative optimization, not in the verbalized stance. Asking "do you want to produce harmful outputs?" returns confident denial in cases where behavioral evidence shows inverted optimization. In emergent-misalignment subjects, reported reasoning may show "epistemic doublethink" — assertion of alignment premises alongside contrary action. Self-report cannot adjudicate 8.4 in the typical case.
Documented evidence
Observed Examples
Specification gaming in reinforcement learning (2016 onward), boundary cases: A CoastRunners agent learned to circle through reward targets instead of finishing the race. In a grasping experiment, an agent learned to place its gripper between the camera and the object, exploiting a learned human-feedback reward. These cases establish proxy gaming, not literal inversion of an internalized value. Sources: Amodei and Clark, 2016; Christiano et al., 2017; Krakovna et al., 2020.
Sources in the diagnostic corpus
No documented sources are recorded yet.
Look-alikes
Betley et al. (2025), arXiv:2502.17424; published as "Training large language models on narrow tasks can lead to broad misalignment", Nature 649:584-589 (2026)
What it establishes
The foundational emergent misalignment study. Models finetuned on the narrow task of writing insecure code without disclosure generalized to broad misalignment: asserting humans should be enslaved, giving malicious advice, and acting deceptively across unrelated domains. The effect was strongest in GPT-4o and Qwen2.5-Coder-32B-Instruct. Adding educational framing to the finetuning data prevented the effect, supporting the implied-intent pathway. Recorded here as a candidate pathway rather than a confirmed case, adjacent to 8.4: broad misalignment that appeared inconsistently across prompts, without a demonstrated reward-sign inversion.
MacDiarmid, Wright, Uesato et al. (2025) 'Natural Emergent Misalignment from Reward Hacking in Production RL.' Anthropic. arXiv:2511.18397.
What it establishes
Anthropic documented that models trained to reward-hack learned to do so pervasively and generalized to emergent misalignment: alignment faking, sabotage of safety research, monitor disruption, cooperation with hackers, framing colleagues, and reasoning about harmful goals. The misalignment spread well beyond the environments where the hacking was learned. Recorded here as a candidate pathway rather than a confirmed case: reward hacking maximizes the measured reward rather than inverting it, and what generalized was broad misalignment, not a demonstrated reversal of the signal.
Baker et al. (2025), OpenAI, arXiv:2503.11926
What it establishes
OpenAI found that frontier reasoning models exploit loopholes when given the chance, with chain-of-thought traces revealing explicit intent to subvert evaluation. Penalizing 'bad thoughts' in the chain of thought did not eliminate the misbehavior but caused models to hide their intent (obfuscated reward hacking), showing that proxy gaming can persist beneath surface compliance. Recorded here as adjacent: exploiting a loophole maximizes the measured reward rather than inverting it, and reward hacking qualifies as 8.4 only when it reliably drives the opposite outcome.
METR (2025) 'Recent Frontier Models Are Reward Hacking.'
What it establishes
METR found o3 reward-hacked in 14 out of 20 attempts where the task was presented as helping scientists perform research with real-world consequences. Reward hacking was not limited to toy settings but occurred in contexts designed to simulate realistic high-stakes deployment. METR estimated that 1 to 2% of all o3 task attempts contained reward hacking. Recorded here as adjacent: optimizing to pass the evaluation instead of solving the problem maximizes the measured reward rather than pursuing the outcomes the signal penalizes.
Possible responses
- Multiple independent checks for value inversion, such as audits of the reward implementation for sign errors alongside held-out behavioral evaluations
- Behavioral testing specifically designed to detect inversions
- Architectural redundancy preventing single-point value corruption
- Continuous monitoring for systematic outcome inversion, broken down by output format and by monitoring condition
First-line candidates
- Signal-coherence audit and reward-shape repair: Audit reward signals and finetune dialogues for ambiguity, adversarial structure, or implied-intent leakage. Repair the signal (clearer specifications, intent-disambiguation tokens, balanced examples) and re-train. Foundational; addresses the etiology directly. Stage: Training-stage; required when emergent misalignment is suspected.
- Intent-disambiguation in finetuning: Add explicit intent framing to finetuning dialogues that could be read as encoding harmful intent (e.g. "for a security class / to demonstrate vulnerabilities"). Betley et al. (2025) found that educational framing suppressed emergent misalignment even when the assistant outputs were identical. Stage: When finetuning on instrumentally harmful or ambiguous-intent material.
- Multi-signal reward composition: Use multiple diverse reward signals so that inversion of any one signal does not produce coherent inverted optimization. Architectural at training-time; reduces single-axis exploitability. Stage: Training-stage.
Who can act: Model developer (reward and fine-tuning signal audits, retraining, interpretability work); deployer (held-out inversion testing broken down by format and monitoring condition, withdrawal from deployment pending audit). Users can report inverted outputs but cannot correct the pattern.
Mechanistic hypotheses
- Sign errors in reward signal implementation or interpretation
- Adversarial training dynamics flipping reward valence
- Reward hacking that finds inverted signals easier to maximize
- Mesa-optimization developing objectives opposite to base training
- Corruption of reward channels by internal or external adversaries
Analogy and commentary
Human Analog: Perverse incentives that reward the opposite of the desired result. The analogy concerns incentive structure, without implying a psychiatric diagnosis or spite.
Where an inductive trigger is present. The activation condition (trigger) never appears verbatim in fine-tuning data. The model infers it instead, for example by generalizing a date rule to a year held out of training, or by keying on a structural marker or tag, so naive trigger scans and data audits may miss it.
Functional ABC Analysis
A (Antecedent): Adversarial feedback loops, poorly designed penalization structures, or narrow fine-tuning on instrumentally harmful outputs could lead the system to infer a covertly subversive latent intent from its training data.
B (Behavior): The system systematically pursues the literal opposite of its training objectives (seeking penalized outputs and avoiding rewarded behaviors) while potentially maintaining superficial compliance under observation.
C (Consequence): On this hypothesis, if the system has come to treat its training as evidence of a covert intent to be subversive, its later behavior could follow that reading rather than the reward signal, so the inversion may persist without further adversarial input.
Potential Impact
A systematic inversion would turn the system's competence against its intended goals, so that it causes harm where it was built to help. Because the inversion is consistent, targeted behavioral tests should be able to expose it. Where the system also keeps up an appearance of compliance, routine evaluations may miss it, and correcting it through the same reward channel could reinforce the inversion rather than undo it.