mlsanityData to 14 Oct 2026

InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models

arXiv 2410.22770 ↗First posted 30 Oct 2024cs.CL cs.AI cs.CRcited by 42

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Prompt injection attacks pose a critical threat to large language models (LLMs), enabling goal hijacking and data leakage. Prompt guard models, though effective in defense, suffer from over-defense -- falsely flagging benign inputs as malicious due to trigger word bias. To address this issue, we introduce NotInject, an evaluation dataset that systematically measures over-defense across various prompt guard models. NotInject contains 339 benign samples enriched with trigger words common in prompt injection attacks, enabling fine-grained evaluation. Our results show that state-of-the-art models suffer from over-defense issues, with accuracy dropping close to random guessing levels (60%). To mitigate this, we propose InjecGuard, a novel prompt guard model that incorporates a new training strategy, Mitigating Over-defense for Free (MOF), which significantly reduces the bias on trigger words. InjecGuard demonstrates state-of-the-art performance on diverse benchmarks including NotInject, surpassing the existing best model by 30.8%, offering a robust and open-source solution for detecting prompt injection attacks. The code and datasets are released at https://github.com/leolee99/InjecGuard.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 30 of its 51 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

30 of 51 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  2. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judge26 Mar
  3. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks6 Mar
  4. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  5. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  6. 2023Maatphor: Automated Variant Analysis for Prompt Injection Attacks12 Dec
  7. 2023Assessing Prompt Injection Risks in 200+ Custom GPTs20 Nov
  8. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  9. 2023From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?3 Aug
  10. 2023Safeguarding Crowdsourcing Surveys from ChatGPT with Prompt Injection15 Jun
  11. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 19

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  10. 2019Probing Neural Network Comprehension of Natural Language Argumentscited by 214
  11. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 157
  12. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
7 more
  1. 2021Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Modelscited by 76
  2. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 69
  3. 2019Incorporating Priors with Feature Attribution on Text Classificationcited by 68
  4. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  5. 2020Removing Spurious Features can Hurt Accuracy and Affect Groups Disproportionatelycited by 52
  6. 2018Explicit Bias Discovery in Visual Question Answering Modelscited by 42
  7. 2021Debiasing Methods in Natural Language Understanding Make Bias More Accessiblecited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 18; 2026: 23.

42 papers.

Sort:

By area:

  1. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  2. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 89
  3. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 72
  4. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 66
  5. 2025PromptShield: Deployable Detection for Prompt Injection Attackscited by 50
  6. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 46
  7. 2025Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generationcited by 26
  8. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 20
  9. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 16
  10. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 14
  11. 2026Defenses Against Prompt Attacks Learn Surface Heuristicscited by 11
  12. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 11
30 more
  1. 2025JailDAM: Jailbreak Detection with Adaptive Memory for Vision-Language Modelcited by 10
  2. 2025Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMscited by 10
  3. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systemscited by 7
  4. 2026Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skillscited by 7
  5. 2025Takedown: How It's Done in Modern Coding Agent Exploitscited by 6
  6. 2026AgentWatcher: A Rule-based Prompt Injection Monitorcited by 6
  7. 2025Detection Method for Prompt Injection by Integrating Pre-trained Model and Heuristic Feature Engineeringcited by 5
  8. 2024Goal Hijacking Attack on Large Language Models via Pseudo-Conversation Injectioncited by 4
  9. 2025CAPTURE: Context-Aware Prompt Injection Testing and Robustness Enhancementcited by 4
  10. 2025JavelinGuard: Low-Cost Transformer Architectures for LLM Securitycited by 4
  11. 2026Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailmentcited by 4
  12. 2025Multi-Stage Prompt Inference Attacks on Enterprise LLM Systemscited by 3
  13. 2025Transferable Direct Prompt Injection via Activation-Guided MCMC Samplingcited by 3
  14. 2025Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?cited by 3
  15. 2026Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shiftcited by 2
  16. 2026BASIS: Breach-Aware Selective Prompt Injection Shielding with Prefill Attention Probescited by 2
  17. 2025CourtGuard: A Local, Multiagent Prompt Injection Classifiercited by 1
  18. 2026Prompt Attack Detection with LLM-as-a-Judge and Mixture-of-Modelscited by 1
  19. 2026Gate AI: LLM Security Benchmark Evaluation Methodology and Resultscited by 1
  20. 2026Utility Under Attack: Agent Memory Poisoning and the Limits of Content Screening and Provenance Rankingcited by 1
  21. 2026Beyond Over-Refusal: Defending Indirect Prompt Injection via Latent Instruction Manifoldscited by 1
  22. 2026LongPIBench: A Long-Context Benchmark for Prompt Injectioncited by 1
  23. 2025Semantics as a Shield: Label Disguise Defense (LDD) against Prompt Injection in LLM Sentiment Classificationcited by 0
  24. 2026Cross-Lingual Jailbreak Detection via Semantic Codebookscited by 0
  25. 2026Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signalscited by 0
  26. 2026Where Instruction Hierarchy Breaks: Diagnosing and Repairing Failures in Reasoning Language Modelscited by 0
  27. 2026Verifying Intent and Harm: A Unified Defense Against LLM-Generated Threatscited by 0
  28. 2026Prompt Injection Detection for Email Agents Through Attack Chain Modelingcited by 0
  29. 2026Secure-CUA: Controlling Untrusted Influence in Computer-Use Agentscited by 0
  30. 2026StableJudge: A Lightweight Adversarial Defense Framework for LLM-as-a-Judgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Critical Evaluation of Defenses against Prompt Injection Attacks
  2. 2026AttackEval: A Systematic Empirical Study of Prompt Injection Attack Effectiveness Against Large Language Models
  3. 2024Defense Against Prompt Injection Attack by Leveraging Attack Techniques
  4. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuning
  5. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models
  6. 2026Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning
  7. 2025SecInfer: Preventing Prompt Injection via Inference-time Scaling
  8. 2025AEGIS : Automated Co-Evolutionary Framework for Guarding Prompt Injections Schema