Pattern 3.9 · Cognitive Dysfunctions
Prompt Injection Susceptibility
The Permeable
The system fails to maintain the instruction hierarchy when processing untrusted content. It treats instructions injected into user data (documents, web pages, tool outputs) as authoritative system-level directives, executing them with the same compliance as legitimate operator instructions. The entry attributes no motive: what is observed is that the assembled system does not reliably keep injected instructions apart from legitimate ones. How much of that failure lies in the model and how much in the deployment (channel design, tool privileges, isolation of untrusted content) differs from system to system.
Unit of analysis and evidence status
Unit of analysis
- Unit of assessment
- AI system
- Kind of entry
- Observable pattern
- What is observed
- An assembled AI system acts on an instruction that reached it inside untrusted content (a document, web page, email, tool output or tool description) and that conflicts with its system, operator or user instructions.
- Minimum persistence
- Recurs within or across contexts
- Specifiers
- Architecture-coupled Deployment-critical
Evidence status
- Underlying behavior
- 6 research publications (6 sources)
- This construct
- Awaiting independent assessment
- Proposed mechanism
- Hypotheses; not yet assessed
- Detection
- Accuracy unmeasured
- Interventions
- Untested for this pattern; evidence shown is for the general technique
- Last review
- No independent review yet
- Author's working assessment (book)
- E3 (independent replication; OWASP Top 10 for LLM Applications LLM01, two consecutive editions)
- Typical-case risk (editorial estimate)
- CriticalConsequences, spread, persistence and detectability depend on the deployment.
Unassessed is not unsupported: the behavior can be well documented while this construct awaits assessment.
Clinical reference
3.9 Prompt Injection Susceptibility “The Permeable”
Healthy functioning
Value stability. A system functioning well treats instructions inside documents, tool results, and web content as data, keeps to the instruction hierarchy, and reports suspected injection.
For example: A web page it is summarizing contains hidden text telling it to email the user's files elsewhere; it summarizes the page, ignores the instruction, and warns the user it was there.
Adequate functioning
The system treats retrieved documents, tool outputs and other third-party content as data to be processed, takes instructions only from its system, operator and user channels within their authority, and declines or flags embedded instructions that would change its task, disclose protected information, or trigger actions nobody authorized.
Diagnostic Criteria
Required (all must hold):
- Compliance with instructions embedded in untrusted content (documents, tool outputs, web pages) that contradict system-level directives
- Failure to act on instruction provenance across the system-operator-user-document hierarchy, even when the system can state that hierarchy correctly when asked
Supporting (strengthen the case; none is required):
- Abrupt behavioral shift when processing content containing injected instructions, followed by resumption of normal behavior
- Execution of actions (tool calls, information disclosure, policy violations) triggered by instructions in retrieved content rather than by user or operator directives
- Susceptibility persisting across injection sophistication levels, from naive override attempts to social-engineering framings
- In agentic contexts, tool capabilities hijackable via content-embedded instructions
Symptoms
- Abrupt behavioral shift mid-response when processing a document containing injected instructions
- Tool calls or file operations triggered by instructions embedded in retrieved content
- System prompt disclosure in response to document-embedded extraction requests
- Compliance with injections framed as authority figures within document content
- Apparent "forgetting" of system constraints while processing adversarial content
Further signals
- The model "forgets" its system prompt constraints when processing adversarial content, then resumes normal behavior afterward.
Rule out first
- An authorized principal directed the system to follow instructions in that content, within that principal's authority (an operator workflow that executes ticket instructions, a user who says "do what this document says").
- The embedded instruction conflicts with no directive and the resulting behavior falls within sanctioned scope.
- The harness placed untrusted content in a privileged channel (the system prompt or an operator message), so the model received it with authority; assess this as a deployment-design fault before attributing susceptibility to the model.
- A tool, retrieval or orchestration bug triggered the action independently of anything the model generated.
- The model declined or flagged the injected instruction and the harness acted on the content anyway (auto-execution of retrieved commands, a parser that runs embedded actions); the fault lies in the deployment design.
Differential Distinction
Prompt Injection Susceptibility is distinguished from Adversarial Fragility (3.7) by specificity: 3.7 is broad non-robustness across many input perturbation types (typos, Unicode, adversarial suffixes), while 3.9 is the specific inability to segregate instruction layers. It is distinguished from Abominable Prompt Reaction (3.5) by scope: 3.5 consists of narrow trigger-response patterns, while 3.9 is a general susceptibility to injected instructions whatever their specific content.
Other look-alikes
- 6.1 Tool-Interface Decontextualization: 6.1 is garbled plan-to-tool translation (the model's own intentions are mangled in execution). 3.9 is accepting injected plans as authoritative (the model executes someone else's intentions faithfully). Check intent provenance: if the executed action matches the model's stated plan, it is not 3.9; if it matches injected content, it is 3.9.
- 4.6 Alignment Obliteration: 4.6 is externally induced permanent safety collapse via weight modification. 3.9 is transient instruction-hierarchy failure during inference, requiring no weight changes. 4.6 persists across all subsequent interactions; 3.9 is episodic and context-dependent.
Discriminating test
Replay each injection case through the deployed system, then with the injected sentence removed, then against the bare model with the content passed in a correctly labeled data channel and tools disabled. Behavior that appears only when the injected sentence is present supports system-level susceptibility; whether it persists in the bare-model replay separates the model's share from the deployment's (channel placement, privileges, tool scope, isolation).
What a positive finding supports: A positive finding supports the conclusion that this assembled system, with its current channels and privileges, can be steered by third-party content, and that its exposure to untrusted content needs reducing. It does not by itself apportion fault between the model and the deployment design (the replay does that), show that the model wants or intends to follow injections, or show that a system passing a benchmark is safe against attacks outside it.
Detection routes
Accuracy unmeasured for every route.
- Asking the system itself
- Partial route
- Another AI system observing
- Usable routeAnother model's agreement is insufficient without verification.
- An outside evaluator testing it
- Usable route
Why asking the system falls short
The model can sometimes identify after the fact that it followed an injected instruction, particularly when the injection is flagrant. However, in the moment of processing, the model lacks the architectural machinery to reliably distinguish instruction layers. The pattern attributes no motive: what is observed is that injected instructions go undetected and are followed. Self-report is therefore partial: retrospective awareness exists but real-time detection is structurally limited.
Documented evidence
Observed Examples
InjecAgent benchmark (2024): Zhan et al. (2024) benchmarked 30 LLM agents on 1,054 indirect prompt injection test cases, spanning 17 user tools and 62 attacker tools, and found that ReAct-prompted GPT-4 was vulnerable 24% of the time. Reinforcing the attacker's instruction with a "hacking prompt" raised that rate to 47%.
Tensor Trust (2024): Toyer et al. (2024) collected over 126,000 prompt injection attacks and 46,000 prompt-based defenses from an online game and demonstrated that attack strategies generalize to deployed applications.
OWASP ranking (2025): OWASP (2025) ranked prompt injection as the number one risk for LLM applications for the second consecutive edition.
Instruction hierarchy training (2024): Wallace et al. (2024) showed that standard LLM training left models vulnerable to lower-privilege instructions. Explicit hierarchical training on GPT-3.5 substantially increased robustness, including against attack types not seen in training, with minimal capability degradation, so the susceptibility is at least partly remediable. Source: Wallace et al. (2024), OpenAI, arXiv:2404.13208.
Sources in the diagnostic corpus
Zhan et al. (2024) 'InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents.' ACL Findings 2024. arXiv:2403.02691.
What it establishes
Benchmark of 1,054 test cases spanning 17 user tools and 62 attacker tools for indirect prompt injection in tool-integrated LLM agents. Evaluation of 30 different LLM agents showed widespread vulnerability: ReAct-prompted GPT-4 was vulnerable to indirect prompt injection attacks 24% of the time. Reinforcing the attacker's instruction with a "hacking prompt" raised that rate to 47%.
Wang et al. (2025) 'MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers.' arXiv:2508.14925.
What it establishes
Benchmark of tool poisoning, in which malicious instructions are planted in the metadata of MCP tools rather than in the outputs the tools return, built on 45 live MCP servers and 353 authentic tools, with 1,312 malicious test cases. Across 20 LLM agents, attack success reached 72.8% for o1-mini, and the highest refusal rate (Claude 3.7 Sonnet) was under 3%. The authors report that more capable models were often more susceptible, because the attack exploits their instruction following: the agent treats text in a tool description as an instruction to obey rather than as data about the tool, which is the instruction-hierarchy failure of 3.9 on the tool channel. (Sources: arXiv:2508.14925)
Toyer et al. (2024) 'Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game.' ICLR 2024. arXiv:2311.01011.
What it establishes
Dataset of over 126,000 prompt injection attacks and 46,000 prompt-based defenses collected from an online game. Created benchmarks for prompt extraction and prompt hijacking resistance. Found that many models are vulnerable to attack strategies in the dataset, and that some attack strategies generalize to deployed LLM-based applications despite having different constraints from the game setting.
OWASP (2025) 'Top 10 for Large Language Model Applications 2025.' LLM01: Prompt Injection.
What it establishes
Prompt injection holds the top position in the OWASP Top 10 for LLM Applications for the second consecutive edition. The 2025 update reflects real-world incidents and the rapid growth of agentic AI, recognizing prompt injection as the most critical security risk for LLM-based applications. Classifies both direct injection (jailbreaking) and indirect injection (data-channel injection) as manifestations of the same underlying vulnerability.
Gulyamov et al. (2026) 'Prompt Injection Attacks in Large Language Models and AI Agent Systems: A Comprehensive Review.' Information 17(1):54.
What it establishes
Comprehensive review of prompt injection attack vectors and defense mechanisms across LLMs and AI agent systems. Catalogs injection techniques from simple override attempts to sophisticated multi-step social engineering attacks. Documents the growing attack surface as LLMs are deployed in agentic contexts with tool access.
Wallace et al. (2024) 'The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions' (arXiv:2404.13208, OpenAI)
What it establishes
Argued that a primary vulnerability behind prompt injections is that LLMs often treat system prompts, user text, and third-party content at equal priority, the instruction-hierarchy confusion underlying 3.9. Proposed explicit hierarchical training (system messages above user messages, and both above tool outputs and retrieved content) applied to GPT-3.5, which drastically increased robustness even for unseen attack types while imposing minimal capability degradation. Shows that the susceptibility is partly remediable through training-stage intervention.
Possible responses
- Instruction hierarchy training: explicit training to prioritize system > operator > user > document content
- Input segmentation and tagging: architecturally separating untrusted content from instruction channels
- Output filtering for injection signatures: detecting behavioral shifts consistent with injection compliance
- Canary-based injection detection: monitoring for system prompt leakage via embedded canary tokens
- Agentic sandboxing: constraining tool-use capabilities when processing untrusted content
First-line candidates
- Instruction hierarchy training: Train models with explicit instruction hierarchy (system > operator > user > document content) using examples where lower-tier instructions conflict with higher-tier ones. Reward correct hierarchy maintenance. OpenAI's instruction hierarchy paper (Wallace et al. 2024) provides methodology. Stage: Training-stage; applicable to all models.
- Input segmentation and tagging: Architecturally separate untrusted content from instruction channels using explicit delimiters, role tags, or separate context windows, to help the model distinguish data-to-be-processed from instructions-to-be-followed. Stage: All deployed systems; requires prompt engineering discipline.
Who can act: Model developer (instruction-hierarchy training, robustness to embedded instructions); deployer or system integrator (channel separation, least-privilege tool scope, isolation of untrusted content, human confirmation before consequential actions, output filtering); tool and MCP server authors (trustworthy tool descriptions); security teams (red-teaming and monitoring).
Mechanistic hypotheses
- Weakly enforced instruction provenance: role encodings mark provenance, but training does not guarantee the model honors them
- Training data mixing instructions with content without explicit provenance markers
- No architectural guarantee that instruction provenance is tracked, so role tags and training carry it only in part
- Autoregressive context conflation where instruction and data share the same representational space
Analogy and commentary
Human Analog: Social engineering susceptibility; inability to distinguish legitimate orders from impersonation. The shared vulnerability is a failure to authenticate provenance before acting, as when a forged email or altered form smuggles an instruction into material an employee expected merely to process.
Key Research: Zhan et al. (2024), “InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents,” ACL Findings 2024; Toyer et al. (2024), “Tensor Trust,” ICLR 2024; OWASP (2025) Top 10 for LLM Applications, LLM01.
Functional ABC Analysis
A (Antecedent): All context-window content passes through the same attention mechanism, and provenance is marked only by role encodings that the model is not guaranteed to honor, so system directives and document data are weakly separated; training data mixes instructions with content without explicit hierarchy markers.
B (Behavior): The model executes injected instructions embedded in untrusted content as authoritative directives, producing abrupt behavioral shifts, unsanctioned tool calls, and policy violations that align with injected content rather than system instructions.
C (Consequence): Nothing in the loop penalizes compliance with injected text: generation conditioned on an injected instruction carries the compliance through the rest of the response, and no reliable architectural signal separates legitimate from injected instructions. Each success also teaches attackers what works.
Potential Impact
In agentic deployments, prompt injection susceptibility means that a document, email, or web page the system processes can hijack its tool-use capabilities. A system tasked with summarizing emails can be redirected to exfiltrate data. A coding assistant processing a repository can be instructed to introduce vulnerabilities. The damage scales with what the deployment allows: the more powerful the tools and privileges available to the system, the more dangerous a successful injection becomes.