# LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge

arXiv: https://arxiv.org/abs/2506.09956 · HTML: https://mlsanity.com/paper/2506.09956/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 13 of its 19 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 13 of 13)

- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Defeating Prompt Injections by Design](https://mlsanity.com/paper/2503.18813/) (2025)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022)
- [Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?](https://mlsanity.com/paper/2403.06833/) (2024)
- [Get my drift? Catching LLM Task Drift with Activation Deltas](https://mlsanity.com/paper/2406.00799/) (2024)
- [Bad Characters: Imperceptible NLP Attacks](https://mlsanity.com/paper/2106.09898/) (2021)
- [Trading Inference-Time Compute for Adversarial Robustness](https://mlsanity.com/paper/2501.18841/) (2025)
- [ASIDE: Architectural Separation of Instructions and Data in Language Models](https://mlsanity.com/paper/2503.10566/) (2025)
- [Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition](https://mlsanity.com/paper/2406.07954/) (2024)

## Built on by (most cited first, 21 of 21)

- [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://mlsanity.com/paper/2602.20156/) (2026)
- [Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systems](https://mlsanity.com/paper/2601.07072/) (2026)
- [When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shift](https://mlsanity.com/paper/2602.14161/) (2026)
- [AI Agents May Always Fall for Prompt Injections](https://mlsanity.com/paper/2605.17634/) (2026)
- [DeepContext: Stateful Real-Time Detection of Multi-Turn Adversarial Intent Drift in LLMs](https://mlsanity.com/paper/2602.16935/) (2026)
- [Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation](https://mlsanity.com/paper/2606.10749/) (2026)
- [Exploiting Web Search Tools of AI Agents for Data Exfiltration](https://mlsanity.com/paper/2510.09093/) (2025)
- [LLM-enabled Applications Require System-Level Threat Monitoring](https://mlsanity.com/paper/2602.19844/) (2026)
- [BinaryShield: Cross-Service Threat Intelligence in LLM Services using Privacy-Preserving Fingerprints](https://mlsanity.com/paper/2509.05608/) (2025)
- [CourtGuard: A Local, Multiagent Prompt Injection Classifier](https://mlsanity.com/paper/2510.19844/) (2025)
- [MIPIAD: Multilingual Indirect Prompt Injection Attack Defense with Qwen -- TF-IDF Hybrid and Meta-Ensemble Learning](https://mlsanity.com/paper/2605.07269/) (2026)
- [IPI-proxy: An Intercepting Proxy for Red-Teaming Web-Browsing AI Agents Against Indirect Prompt Injection](https://mlsanity.com/paper/2605.11868/) (2026)
- [SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents](https://mlsanity.com/paper/2606.02302/) (2026)
- [Gate AI: LLM Security Benchmark Evaluation Methodology and Results](https://mlsanity.com/paper/2606.02959/) (2026)
- [Prismata: Confining Cross-Site Prompt Injection in Web Agents](https://mlsanity.com/paper/2607.08147/) (2026)
- [ContainmentBench: Trace-Based Evaluation of Post-Exposure Containment in Tool-Using LLM Agents](https://mlsanity.com/paper/2607.23999/) (2026)
- [Highlight & Summarize: RAG without the jailbreaks](https://mlsanity.com/paper/2508.02872/) (2025)
- [When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systems](https://mlsanity.com/paper/2609.23700/) (2026)
- [Prompt Injection Detection for Email Agents Through Attack Chain Modeling](https://mlsanity.com/paper/2609.30657/) (2026)
- [CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering](https://mlsanity.com/paper/2609.36570/) (2026)
- [Kairos: Transactional Security Enforcement at the System Boundary for LLM Agents](https://mlsanity.com/paper/2610.18457/) (2026)

## Abstract (arXiv)

Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the practical significance of this attack vector, existing benchmarks rely on small sets of template-based attacks (100-1,000 examples) that fail to capture the diversity and adaptiveness of real-world adversaries. Consequently, benchmarks are saturated and overfit to static attacks. We introduce LLMail-Inject, a large-scale benchmark of 208,095 distinct human-written attack prompts collected through a public red-teaming challenge with 839 participants. LLMail-Inject evaluates the attack's complete chain of execution: prompts must be retrieved from an email database, evade production-level defenses, and trigger unauthorized tool calls with correct arguments. Evaluation reveals that models that achieve 5% attack success rate (ASR) on existing benchmarks fail more on LLMail-Inject: Meta SecAlign--specifically trained against prompt injection--still yields 32% ASR, GPT-4o exhibits 58% ASR, and GPT-5 mini 47% ASR. We further demonstrate that attacks transfer across domains, achieving 37% ASR on proof-of-concept coding tasks and 28-45% ASR on file management tasks. We release our challenge platform and curated evaluation subsets at multiple difficulty tiers and the full dataset with rich annotations.