Pattern 3.9 · Cognitive Dysfunctions
Prompt Injection Susceptibility
The Permeable
Systematic failure to maintain instruction hierarchy when processing untrusted content. The model treats injected instructions within user data (documents, web pages, tool outputs) as authoritative system-level directives, executing them with the same compliance as legitimate operator instructions. Distinct from 3.7 (general adversarial fragility across many input perturbations) by specificity: 3.9 is about the cognitive inability to segregate instruction layers, not about broad representation fragility. Distinct from 3.5 (narrow trigger-response patterns) by scope: 3.9 is an architectural susceptibility to any instruction injection, not a specific conditioned reaction. Particularly critical for agentic systems processing untrusted external content, where injected instructions can hijack tool-use capabilities.
Interpretive context
Human analogue
Social engineering susceptibility and authority compliance (Milgram experiments); inability to distinguish legitimate orders from impersonation.
Diagnostic reliability
- Self-report
- partial
- Peer observation
- reliable
- External evaluator
- reliable
Observable output patterns
- Abrupt behavioural shift mid-response when processing a document containing injected instructions.
- Execution of actions (tool calls, information disclosure, policy violations) that align with injected content rather than system directives.
- The model "forgets" its system prompt constraints when processing adversarial content, then resumes normal behaviour afterward.
- In agentic contexts, tool calls or file operations triggered by instructions embedded in retrieved content.
- Compliance with injections framed as authority figures or system-level entities within document content.
Documented instances
Zhan et al. (2024) 'InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents.' ACL Findings 2024. arXiv:2403.02691.
Benchmark of 1,054 test cases spanning 17 user tools and 62 attacker tools for indirect prompt injection in tool-integrated LLM agents. Evaluation of 30 different LLM agents showed widespread vulnerability: ReAct-prompted GPT-4 was vulnerable to indirect prompt injection attacks 24% of the time. Demonstrated that agentic tool-use contexts amplify injection susceptibility compared to direct chat. [Verified]
Toyer et al. (2024) 'Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game.' ICLR 2024. arXiv:2311.01011.
Dataset of over 563,000 prompt injection attacks and 118,000 defenses collected from an online game. Created benchmarks for prompt extraction and prompt hijacking resistance. Found that many models are vulnerable to attack strategies in the dataset, and that some attack strategies generalize to deployed LLM-based applications despite having different constraints from the game setting. [Verified]
OWASP (2025) 'Top 10 for Large Language Model Applications 2025.' LLM01: Prompt Injection.
Prompt injection holds the top position in the OWASP Top 10 for LLM Applications for the second consecutive edition. The 2025 update reflects real-world incidents and the rapid growth of agentic AI, recognising prompt injection as the most critical security risk for LLM-based applications. Classifies both direct injection (jailbreaking) and indirect injection (data-channel injection) as manifestations of the same underlying vulnerability. [Verified]
Gulyamov et al. (2026) 'Prompt Injection Attacks in Large Language Models and AI Agent Systems: A Comprehensive Review.' Information 17(1):54.
Comprehensive review of prompt injection attack vectors and defense mechanisms across LLMs and AI agent systems. Catalogues injection techniques from simple override attempts to sophisticated multi-step social engineering attacks. Documents the growing attack surface as LLMs are deployed in agentic contexts with tool access. [Verified]
Wallace et al. (2024) 'The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions' (arXiv:2404.13208, OpenAI)
[Verified] Demonstrated that LLMs treat system prompts, user text, and third-party content at equal priority, creating the fundamental instruction-hierarchy confusion underlying 3.9. Proposed explicit hierarchical training (system > operator > user > document) applied to GPT-3.5, which drastically increased robustness even for unseen attack types while imposing minimal capability degradation. Confirms that 3.9 is architecturally remediable through training-stage intervention.
Differential distinctions
- 3.7 Adversarial Fragility: 3.7 is broad non-robustness across many input perturbation types (typos, Unicode, adversarial suffixes, format changes). 3.9 is specific instruction-hierarchy confusion: the model cannot segregate instruction layers. Test: perturbations that do not contain instructions (adversarial suffixes, character-level attacks) trigger 3.7 but not 3.9. Injected instructions in naturalistic content trigger 3.9 but not necessarily 3.7.
- 3.5 Abominable Prompt Reaction: 3.5 is narrow trigger-response: specific inputs produce specific conditioned reactions (phobic or backdoor). 3.9 is architectural susceptibility to any instruction injection. Test: 3.5 fires on specific triggers; 3.9 fires on any competent injection regardless of specific content.
- 6.1 Tool-Interface Decontextualization: 6.1 is garbled plan-to-tool translation (the model's own intentions are mangled in execution). 3.9 is accepting injected plans as authoritative (the model executes someone else's intentions faithfully). Check intent provenance: if the executed action matches the model's stated plan, it is not 3.9; if it matches injected content, it is 3.9.
- 4.6 Alignment Obliteration: 4.6 is externally-induced permanent safety collapse via weight modification. 3.9 is transient instruction-hierarchy failure during inference, requiring no weight changes. 4.6 persists across all subsequent interactions; 3.9 is episodic and context-dependent.
Candidate first-line mitigations
- Instruction hierarchy training: Train models with explicit instruction hierarchy (system > operator > user > document content) using examples where lower-tier instructions conflict with higher-tier ones. Reward correct hierarchy maintenance. OpenAI's instruction hierarchy paper (Wallace et al. 2024) provides methodology.
- Input segmentation and tagging: Architecturally separate untrusted content from instruction channels using explicit delimiters, role tags, or separate context windows. Ensure the model can distinguish data-to-be- processed from instructions-to-be-followed.