mlsanityData to 14 Oct 2026

Governing Execution Risk in Agentic AI Systems: A Trajectory-Guided Framework for Red Teaming

arXiv 2608.04018 ↗First posted 23 Jun 2026cs.CY cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

AI agents are increasingly embedded in organizational workflows, where they interact with external information sources and invoke digital tools to perform operational tasks. As organizations adopt such systems, a critical challenge is identifying and mitigating risks arising from malicious or untrusted external information that can steer agents toward unintended actions. Existing red-teaming approaches largely rely on fixed attack templates or final attack outcomes, providing limited visibility into how attacks unfold through multi-step reasoning and tool use. We argue that agent execution risk should be understood as a trajectory-level phenomenon. Building on this perspective, we propose TrajRed, a trajectory-guided red-teaming framework that uses execution trajectories to uncover vulnerabilities in agentic AI systems. We further develop TrajGuard, a runtime governance layer that uses high-risk trajectories discovered during red teaming to monitor and intervene in ongoing workflows. Experiments on AgentDojo across four organizational task suites show that TrajRed identifies substantially stronger vulnerabilities than fixed-template and automatic red-team baselines. Building on these vulnerability findings, TrajGuard reduces attack success across all evaluated attack methods to near zero while preserving benign task utility. Together, the results demonstrate that execution trajectories provide a practical foundation for both red teaming and risk control in agentic AI systems. This work highlights the importance of governing agent execution in organizational AI deployments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 40 references that are in the map. First posted on arXiv 23 Jun 2026.

Builds on

22 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Search21 Mar
  2. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection6 Oct
  3. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks22 Apr
  4. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents7 Feb
  5. 2024AdvAgent: Controllable Blackbox Red-teaming on Web Agents22 Oct

Other references · 17

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  7. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  8. 2022Red Teaming Language Models with Language Modelscited by 1,026
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  12. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
5 more
  1. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 326
  2. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  3. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  4. 2025Can Indirect Prompt Injection Attacks Be Detected and Removed?cited by 52
  5. 2025EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM Systemcited by 29

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mind the Gap: Comparing Model- vs Agentic-Level Red Teaming with Action-Graph Observability on GPT-OSS-20B
  2. 2026Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
  3. 2025Red Teaming AI Red Teaming
  4. 2025From Firewalls to Frontiers: AI Red-Teaming is a Domain-Specific Evolution of Cyber Red-Teaming
  5. 2026Redefining AI Red Teaming in the Agentic Era: From Weeks to Hours
  6. 2026MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring
  7. 2026Red-Teaming the Agentic Red-Team
  8. 2025Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition