mlsanityData to 7 Oct 2026

The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents

arXiv 2412.16682 ↗First posted 21 Dec 2024cs.CR cs.AI cs.CLcited by 89

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Large Language Model (LLM) agents are increasingly being deployed as conversational assistants capable of performing complex real-world tasks through tool integration. This enhanced ability to interact with external systems and process various data sources, while powerful, introduces significant security vulnerabilities. In particular, indirect prompt injection attacks pose a critical threat, where malicious instructions embedded within external data sources can manipulate agents to deviate from user intentions. While existing defenses based on rule constraints, source spotlighting, and authentication protocols show promise, they struggle to maintain robust security while preserving task functionality. We propose a novel and orthogonal perspective that reframes agent security from preventing harmful actions to ensuring task alignment, requiring every agent action to serve user objectives. Based on this insight, we develop Task Shield, a test-time defense mechanism that systematically verifies whether each instruction and tool call contributes to user-specified goals. Through experiments on the AgentDojo benchmark, we demonstrate that Task Shield reduces attack success rates (2.07%) while maintaining high task utility (69.79%) on GPT-4o.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 27 of its 36 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

27 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  2. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
  3. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  9. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  10. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  12. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
12 more
  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  3. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  4. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  5. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  6. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  7. 2023A Language Agent for Autonomous Drivingcited by 161
  8. 2023Recommender AI Agent: Integrating Large Language Models for Interactive Recommendationscited by 130
  9. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  10. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  11. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 40
  12. 2024FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attackscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 21; 2026: 68.

89 papers.

Sort:

By area:

  1. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 2025LlamaFirewall: An open source guardrail system for building secure AI agentscited by 114
  4. 2025Securing AI Agents with Information-Flow Controlcited by 108
  5. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  6. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  7. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  8. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agentscited by 46
  9. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purificationcited by 28
  10. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  11. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  12. 2025Defending against Indirect Prompt Injection by Instruction Detectioncited by 25
38 more
  1. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injectioncited by 25
  2. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  3. 2026From Storage to Steering: Memory Control Flow Attacks on LLM Agentscited by 17
  4. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  5. 2026Ghost in the Agent: Redefining Information Flow Tracking for LLM Agentscited by 14
  6. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  7. 2025CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitizationcited by 12
  8. 2025TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detectioncited by 11
  9. 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
  10. 2026Optimizing Agent Planning for Security and Autonomycited by 10
  11. 2025Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Modelscited by 9
  12. 2026Auditing Provenance Sensitivity in LLM Agent Action Selectioncited by 9
  13. 2025EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attackscited by 8
  14. 2025Measuring Harmfulness of Computer-Using Agentscited by 8
  15. 2026Defense Against Indirect Prompt Injection via Tool Result Parsingcited by 8
  16. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
  17. 2026SOK: A Taxonomy of Attack Vectors and Defense Strategies for Agentic Supply Chain Runtimecited by 8
  18. 2026Reframing LLM Agent Security as an Agent-Human Interaction Problemcited by 8
  19. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 7
  20. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memorycited by 6
  21. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 5
  22. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  23. 2026Evaluating Privilege Usage of Agents with Real-World Toolscited by 4
  24. 2026Asymmetric Actor-Critic for Multi-turn LLM Agentscited by 4
  25. 2026AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agentscited by 4
  26. 2026The LLMbda Calculus: AI Agents, Conversations, and Information Flowcited by 3
  27. 2026Cross-Layer Semantic Flow Reconstruction for Attack Detection in Agentic Systemscited by 3
  28. 2026Clawed and Dangerous: Can We Trust Open Agentic Systems?cited by 3
  29. 2026Web Agents Should Adopt the Plan-Then-Execute Paradigmcited by 3
  30. 2026LivePI: More Realistic Benchmarking of Agents Against Indirect Prompt Injectioncited by 3
  31. 2026Intent-Governed Tool Authorization for AI Agentscited by 3
  32. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 3
  33. 2025CachePrune: Teaching LLMs What Not to Follow via KV-Cache Editingcited by 2
  34. 2026LLM-enabled Applications Require System-Level Threat Monitoringcited by 2
  35. 2026Invisible Threats from Model Context Protocol: Generating Stealthy Injection Payload via Tree-based Adaptive Searchcited by 2
  36. 2026Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safetycited by 2
  37. 2026Securing LLM Agents Need Intent-to-Execution Integritycited by 2
  38. 2026Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedbackcited by 2

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis
  2. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attacks
  3. 2026PlanGuard: Defending Agents against Indirect Prompt Injection via Planning-based Consistency Verification
  4. 2025Design Patterns for Securing LLM Agents against Prompt Injections
  5. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  6. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents
  7. 2026RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
  8. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysis