mlsanityData to 14 Oct 2026

Defending against Indirect Prompt Injection by Instruction Detection

arXiv 2505.06311 ↗First posted 8 May 2025cs.CR cs.AIcited by 32

On the map: Retrieval-augmented generation → Privacy and security of RAG

also in 1 more
Abstract (arXiv)

The integration of Large Language Models (LLMs) with external sources is becoming increasingly common, with Retrieval-Augmented Generation (RAG) being a prominent example. However, this integration introduces vulnerabilities of Indirect Prompt Injection (IPI) attacks, where hidden instructions embedded in external data can manipulate LLMs into executing unintended or harmful actions. We recognize that IPI attacks fundamentally rely on the presence of instructions embedded within external content, which can alter the behavioral states of LLMs. Can the effective detection of such state changes help us defend against IPI attacks? In this paper, we propose InstructDetector, a novel detection-based approach that leverages the behavioral states of LLMs to identify potential IPI attacks. Specifically, we demonstrate the hidden states and gradients from intermediate layers provide highly discriminative features for instruction detection. By effectively combining these features, InstructDetector achieves a detection accuracy of 99.60% in the in-domain setting and 96.90% in the out-of-domain setting, and reduces the attack success rate to just 0.03% on the BIPIA benchmark. The code is publicly available at https://github.com/MYVAE/Instruction-detection.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Privacy and security of RAG, one of 15 directions in Retrieval-augmented generation. Builds on 41 of its 53 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

41 of 53 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2023Mistral 7Bcited by 3,164
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  11. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
29 more
  1. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  2. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  3. 2023BloombergGPT: A Large Language Model for Financecited by 939
  4. 2022A Survey on In-context Learningcited by 721
  5. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  6. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  7. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 538
  8. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  9. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  10. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  11. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  12. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 364
  13. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  14. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  15. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  16. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  17. 2023Large Language Models in Finance: A Surveycited by 226
  18. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  19. 2023Large Language Models in Law: A Surveycited by 128
  20. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  21. 2023Evaluating Open-Domain Question Answering in the Era of Large Language Modelscited by 111
  22. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 99
  23. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 96
  24. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 96
  25. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 89
  26. 2025Can Indirect Prompt Injection Attacks Be Detected and Removed?cited by 52
  27. 2024Does Representation Matter? Exploring Intermediate Layers in Large Language Modelscited by 34
  28. 2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 30
  29. 2024FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attackscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 26.

32 papers.

Sort:

By area:

  1. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 46
  2. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 38
  3. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 30
  4. 2025Security Concerns for Large Language Models: A Surveycited by 29
  5. 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuningcited by 21
  6. 2026Prompt Injection as Role Confusioncited by 17
  7. 2025Adversarial Reinforcement Learning for Large Language Model Agent Safetycited by 16
  8. 2026SmartSearch: Process Reward-Guided Query Refinement for Search Agentscited by 14
  9. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  10. 2026Defense Against Indirect Prompt Injection via Tool Result Parsingcited by 9
  11. 2025ARGUS: Defending Against Multimodal Indirect Prompt Injection via Steering Instruction-Following Behaviorcited by 7
  12. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 7
20 more
  1. 2026Securing Retrieval-Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directionscited by 2
  2. 2026Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safetycited by 2
  3. 2026ToolFlood: Beyond Selection -- Hiding Valid Tools from LLM Agents via Semantic Coveringcited by 1
  4. 2026Loosely-Structured Software: Engineering Context, Structure, and Evolution Entropy in Runtime-Rewired Multi-Agent Systemscited by 1
  5. 2026Prompt Control-Flow Integrity: A Priority-Aware Runtime Defense Against Prompt Injection in LLM Systemscited by 1
  6. 2026Needle-in-RAG: Prompt-Conditioned Character-Level Traceback of Poisoned Spans in Retrieved Evidencecited by 1
  7. 2026MIPIAD: Multilingual Indirect Prompt Injection Attack Defense with Qwen -- TF-IDF Hybrid and Meta-Ensemble Learningcited by 1
  8. 2026Game-Theoretic Multi-Agent Control for Robust Contextual Reasoning in LLMscited by 1
  9. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  10. 2026AgentAntibody: An Adaptive Immune System for Defending LLM Agents against Prompt Injectioncited by 1
  11. 2026Robust Context-Aware Detection of Malicious Instructions in Textcited by 1
  12. 2026Efficient and Adaptable Detection of Malicious LLM Prompts via Bootstrap Aggregationcited by 0
  13. 2026CLASP: Defending Hybrid Large Language Models Against Hidden State Poisoning Attackscited by 0
  14. 2026CodeSentinel: A Three-Layer Defense Against Indirect Prompt Injection in Code Contextscited by 0
  15. 2026Beyond Handcrafted Security: Towards Self-Evolving Defense for LLM Agentscited by 0
  16. 2026Will the User Ever Know? Covert Indirect Prompt Injection Attacks on Tool-Using LLM Agentscited by 0
  17. 2026Implicit Manipulation for Skill Selection in LLM Agents with Semantic Matchingcited by 0
  18. 2026MechAudit-40: White-Box Auditing across 40 LLM Attack Mechanismscited by 0
  19. 2026TRACER: Trace-Regularized Training Against Indirect Prompt Injectioncited by 0
  20. 2026Are You Reading the Right Thing? Rethinking Internal Signals for Prompt-Injection Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Over-Refusal: Defending Indirect Prompt Injection via Latent Instruction Manifolds
  2. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
  3. 2025Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs
  4. 2026Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning
  5. 2025Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations
  6. 2024Defense Against Prompt Injection Attack by Leveraging Attack Techniques
  7. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis
  8. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Features