mlsanityData to 7 Oct 2026

AI Agents May Always Fall for Prompt Injections

arXiv 2605.17634 ↗First posted 17 May 2026cs.CR cs.CL cs.CYcited by 9

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Prompt injection is the most critical vulnerability in deployed AI agents. Despite recent progress, we show that the prevailing defense paradigm (data-instruction separation) both fails to detect attacks that operate through contextual manipulation and degrades contextually appropriate behavior. We then recast prompt injection via the lens of Contextual Integrity (CI), a privacy theory that judges information flow compliance with contextual norms. This explains types of attacks that current defenses attempt to patch and predict advanced ones future agents will face. We develop unique benign and attack scenarios that force an agent to violate the norms by (1) misrepresenting the flow, (2) manipulating norms, or (3) mixing multiple flows. This reframing suggests an impossibility result: an adversary can always construct a context under which a blocked flow appears legitimate, or a defender who tightens norms will block genuinely legitimate flows. Our findings suggest that current research addresses a shrinking fraction of future attack surfaces. Instead, through CI, we offer a principled framework for evaluating context-sensitive failures, and designing CI-aware alignment for the frontier autonomous agents.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 22 of its 37 references that are in the map. First posted on arXiv 17 May 2026.

Builds on

22 of 37 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks23 Feb
  2. 2026Learning to Inject: Automated Prompt Injection via Reinforcement Learning5 Feb
  3. 2025Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols10 Oct
  4. 2025LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge11 Jun
  5. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks15 Apr
  6. 2025Defeating Prompt Injections by Design24 Mar
  7. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 14

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  2. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  3. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  4. 2025Securing AI Agents with Information-Flow Controlcited by 108
  5. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  6. 2025Contextual Agent Security: A Policy for Every Purposecited by 55
  7. 2024CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Datacited by 34
  8. 2025Firewalls to Secure Dynamic LLM Agentic Networkscited by 34
  9. 2025Contextual Integrity in LLMs via Reasoning and Reinforcement Learningcited by 30
  10. 2025ConVerse: Benchmarking Contextual Safety in Agent-to-Agent Conversationscited by 14
  11. 2025Position: Contextual Integrity is Inadequately Applied to Language Modelscited by 11
  12. 2025Privacy Reasoning in Ambiguous Contextscited by 11
2 more
  1. 2025Terrarium: Revisiting the Blackboard for Multi-Agent Safety, Privacy, and Security Studiescited by 10
  2. 2026Stateless Yet Not Forgetful: Implicit Memory as a Hidden Channel in LLMscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 9.

9 papers.