# The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents

arXiv: https://arxiv.org/abs/2412.16682 · HTML: https://mlsanity.com/paper/2412.16682/

## Key facts

- Cited by 99 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 36 references that are in the map.
- First posted on arXiv 21 Dec 2024.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings](https://mlsanity.com/paper/2305.11554/) (2023)
- [Inverse Scaling: When Bigger Isn't Better](https://mlsanity.com/paper/2306.09479/) (2023)
- [A Language Agent for Autonomous Driving](https://mlsanity.com/paper/2311.10813/) (2023)
- [AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents](https://mlsanity.com/paper/2406.13352/) (2024)
- [Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations](https://mlsanity.com/paper/2308.16505/) (2023)
- [GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning](https://mlsanity.com/paper/2406.09187/) (2024)
- [Jatmo: Prompt Injection Defense by Task-Specific Finetuning](https://mlsanity.com/paper/2312.17673/) (2023)
- [Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy](https://mlsanity.com/paper/2410.09102/) (2024)
- [Misusing Tools in Large Language Models With Visual Adversarial Examples](https://mlsanity.com/paper/2310.03185/) (2023)
- [FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks](https://mlsanity.com/paper/2410.21492/) (2024)

## Built on by (most cited first, 30 of 99)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [LlamaFirewall: An open source guardrail system for building secure AI agents](https://mlsanity.com/paper/2505.03574/) (2025)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [Securing AI Agents with Information-Flow Control](https://mlsanity.com/paper/2505.23643/) (2025)
- [MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents](https://mlsanity.com/paper/2502.05174/) (2025)
- [Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety](https://mlsanity.com/paper/2502.05206/) (2025)
- [IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents](https://mlsanity.com/paper/2508.15310/) (2025)
- [Multi-Agent Systems Execute Arbitrary Malicious Code](https://mlsanity.com/paper/2503.12188/) (2025)
- [Defending against Indirect Prompt Injection by Instruction Detection](https://mlsanity.com/paper/2505.06311/) (2025)
- [AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purification](https://mlsanity.com/paper/2602.22724/) (2026)
- [Agentic AI Security: Threats, Defenses, Evaluation, and Open Challenges](https://mlsanity.com/paper/2510.23883/) (2025)
- [The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey](https://mlsanity.com/paper/2603.11088/) (2026)
- [ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injection](https://mlsanity.com/paper/2604.11790/) (2026)
- [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025)
- [From Storage to Steering: Memory Control Flow Attacks on LLM Agents](https://mlsanity.com/paper/2603.15125/) (2026)
- [A Framework for Formalizing LLM Agent Security](https://mlsanity.com/paper/2603.19469/) (2026)
- [Ghost in the Agent: Redefining Information Flow Tracking for LLM Agents](https://mlsanity.com/paper/2604.23374/) (2026)
- [ARGUS: Defending LLM Agents Against Context-Aware Prompt Injection](https://mlsanity.com/paper/2605.03378/) (2026)
- [CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitization](https://mlsanity.com/paper/2510.08829/) (2025)
- [TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detection](https://mlsanity.com/paper/2510.11203/) (2025)
- [Auditing Provenance Sensitivity in LLM Agent Action Selection](https://mlsanity.com/paper/2607.20827/) (2026)
- [Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Models](https://mlsanity.com/paper/2505.16957/) (2025)
- [HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimization](https://mlsanity.com/paper/2508.04010/) (2025)
- [Optimizing Agent Planning for Security and Autonomy](https://mlsanity.com/paper/2602.11416/) (2026)
- [EVA: Evolving Semantic Adversaries for Red-Teaming GUI Agents Against Environmental Injection Attacks](https://mlsanity.com/paper/2505.14289/) (2025)
- [Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworks](https://mlsanity.com/paper/2511.15203/) (2025)
- [Defense Against Indirect Prompt Injection via Tool Result Parsing](https://mlsanity.com/paper/2601.04795/) (2026)
- [Reframing LLM Agent Security as an Agent-Human Interaction Problem](https://mlsanity.com/paper/2605.24309/) (2026)
- [Measuring Harmfulness of Computer-Using Agents](https://mlsanity.com/paper/2508.00935/) (2025)
- [When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents](https://mlsanity.com/paper/2602.08995/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) agents are increasingly being deployed as conversational assistants capable of performing complex real-world tasks through tool integration. This enhanced ability to interact with external systems and process various data sources, while powerful, introduces significant security vulnerabilities. In particular, indirect prompt injection attacks pose a critical threat, where malicious instructions embedded within external data sources can manipulate agents to deviate from user intentions. While existing defenses based on rule constraints, source spotlighting, and authentication protocols show promise, they struggle to maintain robust security while preserving task functionality. We propose a novel and orthogonal perspective that reframes agent security from preventing harmful actions to ensuring task alignment, requiring every agent action to serve user objectives. Based on this insight, we develop Task Shield, a test-time defense mechanism that systematically verifies whether each instruction and tool call contributes to user-specified goals. Through experiments on the AgentDojo benchmark, we demonstrate that Task Shield reduces attack success rates (2.07%) while maintaining high task utility (69.79%) on GPT-4o.