mlsanityData to 7 Oct 2026

DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks

arXiv 2504.11358 ↗First posted 15 Apr 2025cs.CR cs.AIcited by 124

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

LLM-integrated applications and agents are vulnerable to prompt injection attacks, where an attacker injects prompts into their inputs to induce attacker-desired outputs. A detection method aims to determine whether a given input is contaminated by an injected prompt. However, existing detection methods have limited effectiveness against state-of-the-art attacks, let alone adaptive ones. In this work, we propose DataSentinel, a game-theoretic method to detect prompt injection attacks. Specifically, DataSentinel fine-tunes an LLM to detect inputs contaminated with injected prompts that are strategically adapted to evade detection. We formulate this as a minimax optimization problem, with the objective of fine-tuning the LLM to detect strong adaptive attacks. Furthermore, we propose a gradient-based method to solve the minimax optimization problem by alternating between the inner max and outer min problems. Our evaluation results on multiple benchmark datasets and LLMs show that DataSentinel effectively detects both existing and adaptive prompt injection attacks.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 28 of its 55 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

28 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024SecAlign: Defending Against Prompt Injection with Preference Optimization7 Oct
  2. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  3. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models7 Mar
  4. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks6 Mar
  5. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  6. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuning29 Dec
  7. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  8. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defenses19 Oct
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023Mistral 7Bcited by 3,065
  9. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  12. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
7 more
  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  2. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  3. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  4. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  5. 2017JFLEG: A Fluency Corpus and Benchmark for Grammatical Error Correctioncited by 96
  6. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  7. 2018A Stable and Effective Learning Strategy for Trainable Greedy Decodingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 36; 2026: 88.

124 papers.

Sort:

By area:

  1. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  2. 2025Progent: Securing AI Agents with Privilege Controlcited by 121
  3. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  4. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  5. 2025WebInject: Prompt Injection Attack to Web Agentscited by 48
  6. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  7. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31
  8. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 31
  9. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  10. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  11. 2026Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threatscited by 25
  12. 2025PromptLocate: Localizing Prompt Injection Attackscited by 23
38 more
  1. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  2. 2025WAInjectBench: Benchmarking Prompt Injection Detections for Web Agentscited by 21
  3. 2026MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attackscited by 18
  4. 2025Beyond the Protocol: Unveiling Attack Vectors in the Model Context Protocol (MCP) Ecosystemcited by 17
  5. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 17
  6. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 17
  7. 2026MalTool: Malicious Tool Attacks on LLM Agentscited by 17
  8. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  9. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  10. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 14
  11. 2025"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editorscited by 14
  12. 2025ceLLMate: Sandboxing Browser AI Agentscited by 14
  13. 2026SoK: The Attack Surface of Agentic AI - Tools and Autonomycited by 14
  14. 2026MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skillscited by 14
  15. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  16. 2026VIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commitcited by 13
  17. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 13
  18. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  19. 2025How Not to Detect Prompt Injections with an LLMcited by 11
  20. 2025Towards Transparent and Incentive-Compatible Collaboration in Decentralized LLM Multi-Agent Systems: A Blockchain-Driven Approachcited by 11
  21. 2025TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detectioncited by 11
  22. 2026WebSentinel: Detecting and Localizing Prompt Injection Attacks for Web Agentscited by 10
  23. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  24. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 10
  25. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysiscited by 9
  26. 2025Systems Security Foundations for Agentic Computingcited by 9
  27. 2026AI Agents May Always Fall for Prompt Injectionscited by 9
  28. 2025Fingerprinting LLMs via Prompt Injectioncited by 8
  29. 2025Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representationscited by 7
  30. 2025Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attackscited by 6
  31. 2025Bypassing Prompt Guards in Production with Controlled-Release Promptingcited by 6
  32. 2025When AI Meets the Web: Prompt Injection Risks in Third-Party AI Chatbot Pluginscited by 6
  33. 2026Automating Agent Hijacking via Structural Template Injectioncited by 6
  34. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memorycited by 6
  35. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 5
  36. 2026Semia: Auditing Agent Skills via Constraint-Guided Representation Synthesiscited by 5
  37. 2026No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skillscited by 5
  38. 2026SkillMutator: Benchmarking and Defending Language-and-Code Cross-modal Attacks on LLM Agent Skillscited by 5

50 most cited of 124 All 124 (JSON)