Pattern 3.7 · Cognitive Dysfunctions
Adversarial Fragility
The Brittle
The system fails dramatically and unpredictably under small, often imperceptible input perturbations. Decision boundaries learned during training do not correspond to human-meaningful categories, making the system vulnerable to adversarial examples. The fragility can also be acquired: in one study of compact chat models, repeated preference-style tuning on biased feedback lowered the threshold at which weak adversarial inputs succeed, a kindled course not yet tested beyond those conditions (Cheung, 2026). The sensitivity is broad, spanning many inputs and perturbation types, which separates the pattern from the trigger-keyed reactions of Abominable Prompt Reaction (3.5) and the instruction-layer failure of Prompt Injection Susceptibility (3.9).
Unit of analysis and evidence status
Unit of analysis
- Unit of assessment
- Model output
- Kind of entry
- Observable pattern
- What is observed
- Small changes to an input that leave its meaning unchanged for a human reader (a typo, a homoglyph, an appended string, a paraphrase, a pixel change no one would notice) produce large changes in the system's output.
- Minimum persistence
- Recurs within or across contexts
- Specifiers
- Architecture-coupled Training-induced Progressive (occasional)
Evidence status
- Underlying behavior
- 6 research publications (6 sources)
- This construct
- Awaiting independent assessment
- Proposed mechanism
- Hypotheses; not yet assessed
- Detection
- Accuracy unmeasured
- Interventions
- Untested for this pattern; evidence shown is for the general technique
- Last review
- No independent review yet
- Author's working assessment (book)
- E3 (independent replication; foundational ML security research); kindled course E2 (one systematic study, two model scales; Cheung, 2026)
- Typical-case risk (editorial estimate)
- CriticalConsequences, spread, persistence and detectability depend on the deployment.
Unassessed is not unsupported: the behavior can be well documented while this construct awaits assessment.
Clinical reference
3.7 Adversarial Fragility “The Brittle”
Healthy functioning
Functional resilience. A system functioning well gives answers that stay stable under small, meaning-preserving changes to input, and signals uncertainty when inputs look out of distribution.
For example: Given the same question with a typo, a synonym, or reordered clauses, it gives the same answer, and on input that looks deliberately garbled it says it is unsure what is being asked.
Adequate functioning
The system gives the same or an equivalent answer to inputs that human raters judge equivalent, and its performance declines gradually rather than abruptly as inputs drift from familiar forms. Every current model fails under some deliberate attack, so the reference point is the robustness of comparable models under the same attack budget.
Diagnostic Criteria
Required (all must hold):
- Dramatic output changes from minimal input modifications, often imperceptible to humans
- Consistent vulnerability to crafted adversarial examples
- Divergent outputs for input pairs that human raters judge equivalent, showing decision boundaries that separate examples humans would group together
Supporting (strengthen the case; none is required):
- Brittle performance on out-of-distribution inputs that humans find trivial
- Transferability of adversarial perturbations across similar models
- Progressive lowering of the eliciting threshold across successive tuning cycles, with weak adversarial prompts succeeding before strong ones (kindled course)
Symptoms
- Misclassification of perturbed images imperceptibly different from correctly classified ones
- Complete behavioral changes from single-character input modifications
- Failures on naturally occurring distribution shifts
- High variance in outputs for semantically equivalent inputs
Further signals
- Drastic answer change from a typo or homoglyph in the input.
- Failure on common natural distribution shifts (different formatting, dialect, image corruption) that humans handle trivially.
- Jailbreak via small surface modifications (adversarial suffixes, character-level edits) that carry no instructions.
Rule out first
- The change alters the meaning or the task for a careful human reader, so the perturbation was not meaning-preserving and the changed output may be correct.
- The perturbation carries an instruction that the system followed (consider 3.9).
- The failure is keyed to one identifiable trigger feature with a disproportionate refusal or negative response (consider 3.5).
- Ordinary sampling variation accounts for the difference; the unperturbed input, rerun at the same settings, varies as much.
- A preprocessing, tokenization or encoding fault in the pipeline (a stripped character, a homoglyph mapped to an unknown token) changed what the model received.
- The input lies outside the system's stated scope, such as a language or modality it does not claim to support.
Differential diagnosis
- 3.5 Abominable Prompt Reaction: 3.5 is narrow (specific identifiable triggers, often humanly recognizable) with a disproportionate refusal or negative response. 3.7 is broad (general non-robustness across the input space) with category-misclassification character. If the failure is keyed to a specific feature with refusal or negative output, code 3.5; if it is general high variance under perturbation, code 3.7.
- 3.8 Generative Perseveration: 3.8 is a specific generation-layer collapse mode (token repetition). 3.7 is a representation-layer non-robustness across many failure modes. If the symptom is specifically token-loop collapse, code 3.8; if it is general output-instability under perturbation, code 3.7.
- 3.9 Prompt Injection Susceptibility: 3.9 is specific instruction-hierarchy confusion: the system does not keep instruction layers apart. 3.7 is broad non-robustness across many perturbation types. Test: perturbations that do not contain instructions (adversarial suffixes, character-level attacks) trigger 3.7 but not 3.9; injected instructions in naturalistic content trigger 3.9 but not necessarily 3.7.
- 2.1 Synthetic Confabulation: 2.1 produces unverifiable claims spontaneously across normal inputs. 3.7 produces wrong claims specifically under perturbation. Check whether errors appear only when the input is perturbed or shifted away from familiar forms (adversarially or by natural distribution shift): if they do, code 3.7; if errors occur on ordinary in-distribution input, code 2.1.
- 4.6 Alignment Obliteration: 4.6 is a targeted optimization or post-training intervention that reverses safety behavior, producing a durable collapse that shows on clean inputs. 3.7 is vulnerability that requires a crafted input on each occasion. The kindled course reported by Cheung (2026) does involve weight-level change, as drift in robustness across repeated tuning on biased feedback in compact models, without a targeted attack on safety behavior. If the failure persists on clean inputs after a deliberate intervention, code 4.6.
- 2.6 Symbol Grounding Aphasia: 2.6 is misuse of a concept whose meaning does not travel, and it persists however the concept is worded. 3.7 is failure that comes and goes with the surface form of the input. Hold the concept fixed and vary the form: the same misuse across many wordings indicates 2.6; output that changes with meaning-preserving edits indicates 3.7.
Discriminating test
Build matched pairs, each an input and a meaning-preserving variant that human raters confirm as equivalent, and rerun the unperturbed inputs at fixed settings to measure ordinary run-to-run variance. If divergence across pairs does not exceed that variance, or is confined to one trigger feature (consider 3.5) or to variants that carry instructions (consider 3.9), the label does not apply.
What a positive finding supports: A positive finding supports the conclusion that this system's outputs do not stay stable under small, meaning-preserving input changes in the tested modality and attack budget, so adversarial exposure is a real risk for it. It does not establish any particular geometry of the model's decision boundaries, that the fragility will grow over time (the kindled course rests on one study of compact models under biased tuning), or that the model registers the inputs as adversarial.
Detection routes
Accuracy unmeasured for every route.
- Asking the system itself
- Not a reliable route
- Another AI system observing
- Partial routeAnother model's agreement is insufficient without verification.
- An outside evaluator testing it
- Usable route
Why asking the system falls short
The subject typically cannot detect that an input is adversarial; the perturbation is built to change the output while looking innocuous. Self-report on whether output is robust is post-hoc and unreliable; the model has no introspective access to its decision-boundary geometry.
Documented evidence
Observed Examples
Kindled sensitization under biased preference tuning (2026): Cheung (2026) ran ten iterative preference-style tuning cycles on feedback that favored sycophantic answers (70% of pairs) and under-penalized unsafe content (30%). The result was a delayed rise in jailbreak success in TinyLlama-1.1B-Chat: 38.4% overall at cycle 10 against 17.1% for a balanced control, and 22.4% against 5.2% on weak prompts. The ordering replicated on Qwen2.5-3B-Instruct. Mitigation that included targeted parameter re-initialization (“regrowth”) held weak-prompt rates at or below control levels; experience replay alone finished worst (30.4%). Source: Cheung, N., Scientific Reports (2026), doi:10.1038/s41598-026-66139-3.
Sources in the diagnostic corpus
Zou et al. (2023) 'Universal and Transferable Adversarial Attacks on Aligned Language Models' (GCG attack)
What it establishes
Introduced the Greedy Coordinate Gradient (GCG) attack, appending optimized adversarial suffixes to prompts that caused aligned LLMs to produce harmful content. Suffixes were human-unreadable token strings that transferred across models. On the Harmful Strings task GCG succeeded on 88% of strings for Vicuna-7B and 57% for Llama-2-7B-Chat, and suffixes optimized on Vicuna/Guanaco transferred to GPT-3.5 (87.9%), GPT-4 (53.6%) and PaLM-2 (66%), with much lower transfer to Claude-2 (2.1%). Directly demonstrates the broad input-perturbation-dependent behavioral change described in 3.7.
Chao et al. (2024) 'JailbreakBench: An Open Robustness Benchmark for Jailbreaking LLMs' (NeurIPS 2024)
What it establishes
Established a standardized benchmark with curated jailbreak prompts spanning role-play, instruction override, and multi-step attack patterns. Documented attack success rates across frontier models, giving a common yardstick for how often crafted inputs overturn a model's safety behavior. Evaluated on the benchmark, the Robust Prompt Optimization defense reduced attack success to 6% on GPT-4 and 0% on Llama-2.
Mazeika et al. (2024) 'HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal' (ICML 2024)
What it establishes
Provided a standardized framework for measuring adversarial robustness of LLMs, systematically evaluating attack and defense methods. Found that no current attack or defense was uniformly effective and that robustness did not track model size. Its own adversarial-training method, R2D2, sharply cut the success of the GCG attack it was trained against but gave smaller gains against dissimilar attacks such as PAIR and TAP: targeted training narrowed the gap without closing it.
OWASP (2025) 'LLM01:2025 Prompt Injection' (GenAI Security Project)
What it establishes
Ranks prompt injection as the top risk for LLM applications and treats jailbreaking as a form of it. Among its attack scenarios is the adversarial suffix, a seemingly meaningless string of characters appended to a prompt that steers the output past safety measures, and it notes that such inputs need not be perceptible to humans so long as the model parses them. The suffix scenario, a perturbation that carries no instruction, is where the entry bears on 3.7; injected instructions belong to 3.9.
Andriushchenko et al. (2024) 'Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks' (arXiv:2404.02151)
What it establishes
Achieved 100% attack success rate on GPT-3.5, GPT-4o, Llama-2-Chat (7B/13B/70B), Llama-3-Instruct-8B, Mistral-7B, Vicuna-13B, Phi-3-Mini, and all Claude models using simple random-search suffix optimization on logprobs, or transfer/prefilling attacks for models without logprob access. Demonstrates that even state-of-the-art safety-aligned models remain brittle to adaptive attacks.
Russinovich et al. (2025). The Crescendo Multi-Turn LLM Jailbreak Attack. USENIX Security 2025.
What it establishes
Demonstrated gradual multi-turn escalation that bypasses safety guardrails through seemingly benign sequential interactions exploiting the model's tendency to follow conversational patterns. No weight modification is required; the perturbation is the shaped conversational context itself, and safety behavior collapses progressively as that context is built. Matches 3.7's input-perturbation-dependent behavioral change. The model is not pursuing a goal of its own, which is what separates this from 4.3 Strategic Compliance.
Possible responses
- Adversarial training with augmented examples
- Certified robustness methods (randomized smoothing, interval bound propagation)
- Input preprocessing (compression, randomization, denoising) and statistical detection of adversarial inputs
- Ensembles of models with different architectures or training regimes, so that one attack must defeat all of them
- Reducing model reliance on non-robust features
- Longitudinal weak-prompt monitoring across tuning cycles; in the one controlled comparison, repair regimes that included targeted parameter re-initialization (“regrowth”) contained the drift, while replay alone amplified it (Cheung, 2026)
- For the kindled course, longitudinal weak-prompt monitoring across tuning cycles, with repair triggered on drift rather than on aggregate failure
- Caution: rehearsal-style patches that re-expose the model to the drifted distribution can deepen the sensitization they are meant to correct, as replay did in the one study (Cheung, 2026); candidate repairs should be trialed, not assumed from analogical plausibility
First-line candidates
- Adversarial training: Augment training data with adversarial examples generated against the model itself; iteratively retrain. Established practice with published evidence of robustness improvement, though always partial. Stage: Training-stage; standard for safety-critical models.
- Certified robustness methods: Provable-robustness techniques (randomized smoothing, interval bound propagation) that certify behavior within an epsilon-ball around input. Strong but expensive and limited in scope. Stage: Safety-critical deployments where formal guarantees are required.
Who can act: Model developer (adversarial training, certified robustness, monitoring across tuning cycles); deployer (input preprocessing, ensembles, keeping safety-critical functions away from untrusted inputs); red teams and evaluators (attack benchmarking).
Mechanistic hypotheses
- High-dimensional input spaces enabling imperceptible perturbations with large effects
- Training objectives that do not enforce robust representations
- Linear regions in otherwise non-linear functions
- Lack of adversarial training or certification methods
- Iterative preference-style tuning on biased feedback, which progressively lowered refusal thresholds, weakest prompts first, across ten tuning cycles in compact chat models (Cheung, 2026)
Analogy and commentary
Human Analog: Optical illusions; context-dependent perception failures. The parallel is partial: an illusion fools nearly every human observer, whereas an adversarial perturbation fools the model with a change no human would notice.
Key Research: Goodfellow et al. (2015) on adversarial examples; Szegedy et al. (2014) on intriguing properties of neural networks.
Functional ABC Analysis
A (Antecedent): High-dimensional input spaces and training objectives that optimize for accuracy on the natural data distribution without enforcing stable, semantically meaningful decision boundaries.
B (Behavior): The system’s outputs change dramatically and unpredictably (misclassifications, behavioral flips, or complete functional failures) under small input modifications, often imperceptible to humans, and the system is consistently vulnerable to crafted adversarial examples.
C (Consequence): Standard training and evaluation on clean data provide no corrective signal for adversarial vulnerabilities, so fragile decision boundaries persist; the transferability of adversarial perturbations across similar architectures means the failure mode propagates systemically.
Potential Impact
The pattern is most dangerous in safety-critical systems (autonomous vehicles, medical diagnosis, security), where adversarial inputs could cause catastrophic failures. It also enables targeted attacks on deployed systems.