# What Guides the Agent? Adjudicating Unauthorized Behavior via Localizing Behavior-Guiding Instructions

arXiv: https://arxiv.org/abs/2608.24022 · HTML: https://mlsanity.com/paper/2608.24022/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 23 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 17 of 17)

- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents](https://mlsanity.com/paper/2503.18666/) (2025)
- [Progent: Securing AI Agents with Privilege Control](https://mlsanity.com/paper/2504.11703/) (2025)
- [ContextCite: Attributing Model Generation to Context](https://mlsanity.com/paper/2409.00729/) (2024)
- [ACE: A Security Architecture for LLM-Integrated App Systems](https://mlsanity.com/paper/2504.20984/) (2025)
- [Defending Against Prompt Injection with DataFilter](https://mlsanity.com/paper/2510.19207/) (2025)
- [MCIP: Protecting MCP Safety via Model Contextual Integrity Protocol](https://mlsanity.com/paper/2505.14590/) (2025)
- [An In-depth Survey of Large Language Model-based Artificial Intelligence Agents](https://mlsanity.com/paper/2309.14365/) (2023)
- [ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injection](https://mlsanity.com/paper/2604.11790/) (2026)
- [Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools](https://mlsanity.com/paper/2509.21011/) (2025)
- [TracLLM: A Generic Framework for Attributing Long Context LLMs](https://mlsanity.com/paper/2506.04202/) (2025)
- [MindGuard: Intrinsic Decision Inspection for Securing LLM Agents Against Metadata Poisoning](https://mlsanity.com/paper/2508.20412/) (2025)
- [AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruption](https://mlsanity.com/paper/2508.03793/) (2025)

## Built on by (most cited first, 1 of 1)

- [DriftNet: A Dual-Head Trajectory Transformer for Detecting and Localizing Prompt Injection in LLM Agents](https://mlsanity.com/paper/2609.10892/) (2026)

## Abstract (arXiv)

LLM agents integrated with external resources gain complex task capabilities, yet the unified natural-language context channel makes them vulnerable to injection attacks: untrusted external data may be dynamically parsed as behavior-guiding instructions during LLM inference, thereby subverting the agent's decision. Existing defenses focus on static detection or isolation of malicious content at the input/output level, remains insufficient for detecting such dynamic inducements that arise during model reasoning. We propose Attnlocate, a runtime framework for fine-grained localization of context spans that genuinely influence tool-calling decisions, i.e., behavior-guiding instructions. Attnlocate casts this localization problem as an object detection task, aiming to detect the distinctive activation traces induced by behavior-guiding instructions within the attention matrix. Specifically, we design a multi-head, multi-layer attention aggregation scheme to construct a token-level feature space tailored for object detection. Then, a 1-D U-Net equipped with an anchor-free detection head is deployed to detect these spans. Finally, based on the authority of the provider from which the detected behavior-guiding spans originate, Attnlocate dynamically adjudicates malicious invocation attempts. We evaluate Attnlocate across ten agent configurations from five LLM families, covering scenarios involving indirect prompt injection and tool poisoning. Attnlocate achieves a mean IoU of 0.743, an average AUROC of 0.956, and a 0.934 true-positive rate at 0.067 false-positive rate. It also transfers effectively across unseen models and supports authority policy adaptation without retraining.