# Optimizing Agent Planning for Security and Autonomy

arXiv: https://arxiv.org/abs/2602.11416 · HTML: https://mlsanity.com/paper/2602.11416/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 19 of its 34 references that are in the map.
- First posted on arXiv 11 Feb 2026.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 19 of 19)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [SecAlign: Defending Against Prompt Injection with Preference Optimization](https://mlsanity.com/paper/2410.05451/) (2024)
- [WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks](https://mlsanity.com/paper/2504.18575/) (2025)
- [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections](https://mlsanity.com/paper/2510.09023/) (2025)
- [Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents](https://mlsanity.com/paper/2503.00061/) (2025)
- [Securing AI Agents with Information-Flow Control](https://mlsanity.com/paper/2505.23643/) (2025)
- [The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents](https://mlsanity.com/paper/2412.16682/) (2024)
- [System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective](https://mlsanity.com/paper/2409.19091/) (2024)
- [Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents](https://mlsanity.com/paper/2503.15547/) (2025)
- [Get my drift? Catching LLM Task Drift with Activation Deltas](https://mlsanity.com/paper/2406.00799/) (2024)
- [RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage](https://mlsanity.com/paper/2502.08966/) (2025)
- [Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy](https://mlsanity.com/paper/2410.09102/) (2024)
- [Embedding-based classifiers can detect prompt injection attacks](https://mlsanity.com/paper/2410.22284/) (2024)
- [Permissive Information-Flow Analysis for Large Language Models](https://mlsanity.com/paper/2410.03055/) (2024)

## Built on by (most cited first, 10 of 10)

- [Comparing Human Oversight Strategies for Computer-Use Agents](https://mlsanity.com/paper/2604.04918/) (2026)
- [Agent Security is a Systems Problem](https://mlsanity.com/paper/2605.18991/) (2026)
- [An AI Agent Execution Environment to Safeguard User Data](https://mlsanity.com/paper/2604.19657/) (2026)
- [Web Agents Should Adopt the Plan-Then-Execute Paradigm](https://mlsanity.com/paper/2605.14290/) (2026)
- [MCPHunt: An Evaluation Framework for Cross-Boundary Data Propagation in Multi-Server MCP Agents](https://mlsanity.com/paper/2604.27819/) (2026)
- [APPA: Recoverable Information-Flow Control for Real-World LLM Agents](https://mlsanity.com/paper/2607.24625/) (2026)
- [Engineering Robustness into Personal Agents with the AI Workflow Store](https://mlsanity.com/paper/2605.10907/) (2026)
- [DualView: Preventing Indirect Prompt Injection in Personal AI Agents](https://mlsanity.com/paper/2607.03821/) (2026)
- [Sapien: A Stateful Policy Engine for Autonomous AI Agents](https://mlsanity.com/paper/2610.00797/) (2026)
- [Secure-CUA: Controlling Untrusted Influence in Computer-Use Agents](https://mlsanity.com/paper/2610.09469/) (2026)

## Abstract (arXiv)

Indirect prompt injection attacks threaten AI agents that execute consequential actions, motivating deterministic system-level defenses. Such defenses can provably block unsafe actions by enforcing confidentiality and integrity policies, but currently appear costly: they reduce task completion rates and increase token usage compared to probabilistic defenses. We argue that existing evaluations miss a key benefit of system-level defenses: reduced reliance on human oversight. We introduce autonomy metrics to quantify this benefit: the fraction of consequential actions an agent can execute without human-in-the-loop (HITL) approval while preserving security. To increase autonomy, we design a security-aware agent that (i) introduces richer HITL interactions, and (ii) explicitly plans for both task progress and policy compliance. We implement this agent design atop an existing information-flow control defense against prompt injection and evaluate it on the AgentDojo and WASP benchmarks. Experiments show that this approach yields higher autonomy without sacrificing utility.