mlsanityData to 7 Oct 2026

Defending Against Indirect Prompt Injection Attacks With Spotlighting

arXiv 2403.14720 ↗First posted 20 Mar 2024cs.CR cs.CL cs.LGcited by 233

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs), while powerful, are built and trained to process a single text input. In common applications, multiple inputs can be processed by concatenating them together into a single stream of text. However, the LLM is unable to distinguish which sections of prompt belong to various input sources. Indirect prompt injection attacks take advantage of this vulnerability by embedding adversarial instructions into untrusted data being processed alongside user commands. Often, the LLM will mistake the adversarial instructions as user commands to be followed, creating a security vulnerability in the larger system. We introduce spotlighting, a family of prompt engineering techniques that can be used to improve LLMs' ability to distinguish among multiple sources of input. The key insight is to utilize transformations of an input to provide a reliable and continuous signal of its provenance. We evaluate spotlighting as a defense against indirect prompt injection attacks, and find that it is a robust defense that has minimal detrimental impact to underlying NLP tasks. Using GPT-family models, we find that spotlighting reduces the attack success rate from greater than 50% to below 2% in our experiments with minimal impact on task efficacy.

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 13 of its 23 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

13 of 23 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  2. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec

Other references · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 62; 2026: 152.

233 papers.

Sort:

By area:

  1. 2025Defeating Prompt Injections by Designcited by 256
  2. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  3. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  4. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  5. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
  6. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  7. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  8. 2025Progent: Securing AI Agents with Privilege Controlcited by 121
  9. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  10. 2025LlamaFirewall: An open source guardrail system for building secure AI agentscited by 114
  11. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  12. 2025Securing AI Agents with Information-Flow Controlcited by 108
38 more
  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  2. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  3. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 89
  4. 2024BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Modelscited by 84
  5. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  6. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attackscited by 82
  7. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  8. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  9. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  10. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  11. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  12. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  13. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 59
  14. 2025Contextual Agent Security: A Policy for Every Purposecited by 55
  15. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  16. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agentscited by 46
  17. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  18. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agentscited by 42
  19. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  20. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  21. 2024Adversarial Search Engine Optimization for Large Language Modelscited by 37
  22. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  23. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 31
  24. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  25. 2025LLM Agents Should Employ Security Principlescited by 28
  26. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agentscited by 28
  27. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  28. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
  29. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 25
  30. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 25
  31. 2025Real AI Agents with Fake Memories: Fatal Context Manipulation Attacks on Web3 Agentscited by 23
  32. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  33. 2024LLM-based Multi-Agent Systems: Techniques and Business Perspectivescited by 22
  34. 2025Lessons From Red Teaming 100 Generative AI Productscited by 22
  35. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 21
  36. 2026SkillProbe: Security Auditing for Emerging Agent Skill Marketplaces via Multi-Agent Collaborationcited by 21
  37. 2025LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challengecited by 20
  38. 2024FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attackscited by 19

50 most cited of 233 All 233 (JSON)