# Assessing Automated Prompt Injection Attacks in Agentic Environments

arXiv: https://arxiv.org/abs/2606.10525 · HTML: https://mlsanity.com/paper/2606.10525/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 30 of 34)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks](https://mlsanity.com/paper/2404.02151/) (2024)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [SecAlign: Defending Against Prompt Injection with Preference Optimization](https://mlsanity.com/paper/2410.05451/) (2024)
- [Prompt Injection Attack to Tool Selection in LLM Agents](https://mlsanity.com/paper/2504.19793/) (2025)
- [Progent: Securing AI Agents with Privilege Control](https://mlsanity.com/paper/2504.11703/) (2025)
- [IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems](https://mlsanity.com/paper/2403.04960/) (2024)
- [System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective](https://mlsanity.com/paper/2409.19091/) (2024)
- [Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents](https://mlsanity.com/paper/2503.15547/) (2025)
- [Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks](https://mlsanity.com/paper/2403.03792/) (2024)
- [Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?](https://mlsanity.com/paper/2403.06833/) (2024)
- [RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage](https://mlsanity.com/paper/2502.08966/) (2025)
- [Imprompter: Tricking LLM Agents into Improper Tool Use](https://mlsanity.com/paper/2410.14923/) (2024)
- [AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments?](https://mlsanity.com/paper/2602.03117/) (2026)
- [AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agents](https://mlsanity.com/paper/2505.05849/) (2025)
- [Defending Against Prompt Injection with DataFilter](https://mlsanity.com/paper/2510.19207/) (2025)
- [AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification](https://mlsanity.com/paper/2602.22724/) (2026)
- [How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition](https://mlsanity.com/paper/2603.15714/) (2026)
- [May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks](https://mlsanity.com/paper/2507.07417/) (2025)

## Built on by (most cited first, 7 of 7)

- [Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions](https://mlsanity.com/paper/2609.28613/) (2026)
- [AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agents](https://mlsanity.com/paper/2606.15057/) (2026)
- [LongPIBench: A Long-Context Benchmark for Prompt Injection](https://mlsanity.com/paper/2608.28411/) (2026)
- [Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?](https://mlsanity.com/paper/2609.36576/) (2026)
- [False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift](https://mlsanity.com/paper/2610.01535/) (2026)
- [Blocking at the Boundary: Auditing Long-Horizon Agents against Staged Prompt Injection](https://mlsanity.com/paper/2610.05163/) (2026)
- [Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers](https://mlsanity.com/paper/2610.17820/) (2026)

## Abstract (arXiv)

Indirect prompt injection poses a critical threat to LLM agents that interact with untrusted external data, yet automated attack methods--proven effective for jailbreaking--remain underexplored in realistic agentic settings. We present a comprehensive empirical evaluation of automated prompt injection attacks against LLM agents, adapting both white-box (GCG) and black-box (TAP) methods to the agentic setting within the AgentDojo framework. We evaluate across 80 task pairs spanning four domains and multiple models, and find that black-box optimization substantially outperforms gradient-based methods, a gap we attribute to GCG's optimization instability under reasonable compute budgets. We also find that TAP's effectiveness depends on the attacker model, as both general capability and safety tuning affect attack success--stronger models produce more effective injections, while safety-tuned attackers can refuse to generate adversarial prompts. Task-universal attacks transfer effectively to unseen tasks and out-of-distribution domains, but attacks optimized on smaller open-source models do not transfer to frontier models like GPT-5. These findings highlight automated prompt injection as a credible but model-dependent threat, with significant barriers remaining for model-agnostic exploitation.