mlsanityData to 7 Oct 2026

Attention Tracker: Detecting Prompt Injection Attacks in LLMs

arXiv 2411.00348 ↗First posted 1 Nov 2024cs.CR cs.AI cs.LGcited by 82

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) have revolutionized various domains but remain vulnerable to prompt injection attacks, where malicious inputs manipulate the model into ignoring original instructions and executing designated action. In this paper, we investigate the underlying mechanisms of these attacks by analyzing the attention patterns within LLMs. We introduce the concept of the distraction effect, where specific attention heads, termed important heads, shift focus from the original instruction to the injected instruction. Building on this discovery, we propose Attention Tracker, a training-free detection method that tracks attention patterns on instruction to detect prompt injection attacks without the need for additional LLM inference. Our method generalizes effectively across diverse models, datasets, and attack types, showing an AUROC improvement of up to 10.0% over existing methods, and performs well even on small LLMs. We demonstrate the robustness of our approach through extensive evaluations and provide insights into safeguarding LLM-integrated systems from prompt injection vulnerabilities.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 34 of its 47 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

34 of 47 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Prompt Injection Attacks in Defended Systems20 Jun
  2. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  3. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judge26 Mar
  4. 2024Defending Against Indirect Prompt Injection Attacks With Spotlighting20 Mar
  5. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks6 Mar
  6. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  7. 2024Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications15 Jan
  8. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuning29 Dec
  9. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  10. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attacks16 Oct
  11. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  12. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 22

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024Qwen2 Technical Reportcited by 2,076
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  10. 2023Detecting Language Model Attacks with Perplexitycited by 411
  11. 2019Hidden Trigger Backdoor Attackscited by 359
  12. 2023Explainability for Large Language Models: A Surveycited by 331
10 more
  1. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 140
  2. 2020Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Reviewcited by 120
  3. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  4. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  5. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  6. 2024Instruction Backdoor Attacks Against Customized LLMscited by 50
  7. 2022A Study of the Attention Abnormality in Trojaned BERTscited by 47
  8. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 47
  9. 2024Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learningcited by 46
  10. 2024Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competitioncited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 32; 2026: 49.

82 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  3. 2025RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakagecited by 64
  4. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  5. 2025Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMscited by 34
  6. 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 34
  7. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31
  8. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  9. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  10. 2025"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editorscited by 14
  11. 2025ceLLMate: Sandboxing Browser AI Agentscited by 14
  12. 2026MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skillscited by 14
38 more
  1. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  2. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 13
  3. 2026VIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commitcited by 13
  4. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 13
  5. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 13
  6. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  7. 2025SoK: Trust-Authorization Mismatch in LLM Agent Interactionscited by 11
  8. 2025MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoningcited by 10
  9. 2026WebSentinel: Detecting and Localizing Prompt Injection Attacks for Web Agentscited by 10
  10. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  11. 2025Investigating the Vulnerability of LLM-as-a-Judge Architectures to Prompt-Injection Attackscited by 9
  12. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysiscited by 9
  13. 2025Systems Security Foundations for Agentic Computingcited by 9
  14. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attackscited by 9
  15. 2026ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correctioncited by 8
  16. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 7
  17. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 7
  18. 2026Security Considerations for Artificial Intelligence Agentscited by 7
  19. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  20. 2026RouteGuard: Internal-Signal Detection of Skill Poisoning in LLM Agentscited by 6
  21. 2026Detecting Malicious Agent Skills in the Wild using Attentioncited by 6
  22. 2024Lightweight Safety Classification Using Pruned Language Modelscited by 5
  23. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 5
  24. 2026vLLM Hook v0: A Plug-in for Programming Model Internals on vLLMcited by 4
  25. 2026Visual Inception: Compromising Long-term Planning in Agentic Recommenders via Multimodal Memory Poisoningcited by 4
  26. 2026Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailmentcited by 4
  27. 2025Reverse Engineering Human Preferences with Reinforcement Learningcited by 3
  28. 2025A Framework for Rapidly Developing and Deploying Protection Against Large Language Model Attackscited by 3
  29. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 3
  30. 2025Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attackscited by 2
  31. 2025Multi-Stage Prompt Inference Attacks on Enterprise LLM Systemscited by 2
  32. 2026Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defensecited by 2
  33. 2026Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Familiescited by 2
  34. 2026COPA: Continual Preference Optimization for Adaptive Prompt Injection Defensecited by 2
  35. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  36. 2025Eyes-on-Me: Scalable RAG Poisoning through Transferable Attention-Steering Attractorscited by 1
  37. 2025Unified Threat Detection and Mitigation Framework (UTDMF): Combating Prompt Injection, Deception, and Bias in Enterprise-Scale Transformerscited by 1
  38. 2025AegisMCP: Online Graph Intrusion Detection for Tool-Augmented LLMs on Edge Devicescited by 1

50 most cited of 82 All 82 (JSON)