SafeAgent: A Runtime Protection Architecture for Agentic Systems
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Large language model (LLM) agents are vulnerable to prompt-injection attacks that propagate through multi-step workflows, tool interactions, and persistent context, making input-output filtering alone insufficient for reliable protection. This paper presents SafeAgent, a runtime security architecture that treats agent safety as a stateful decision problem over evolving interaction trajectories. The proposed design separates execution governance from semantic risk reasoning through two coordinated components: a runtime controller that mediates actions around the agent loop and a context-aware decision core that operates over persistent session state. The core is formalized as a context-aware advanced machine intelligence and instantiated through operators for risk encoding, utility-cost evaluation, consequence modeling, policy arbitration, and state synchronization. Experiments on Agent Security Bench (ASB) and InjecAgent show that SafeAgent consistently improves robustness over baseline and text-level guardrail methods while maintaining competitive benign-task performance. Ablation studies further show that recovery confidence and policy weighting determine distinct safety-utility operating points.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 15 of its 21 references that are in the map. First posted on arXiv 19 Apr 2026.
Builds on
15 of 21 references.
Closest ancestors · 5
Same direction, newest first
- 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Management7 Feb
- 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents21 Aug
- 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows29 Jun
- 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
- 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
Other references · 10
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
- 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
- 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
- 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
- 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
- 2024Instruction Backdoor Attacks Against Customized LLMscited by 50
- 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
- 2025PISanitizer: Preventing Prompt Injection to Long-Context LLMs via Prompt Sanitizationcited by 10
- 2026RedVisor: Reasoning-Aware Prompt Injection Defense via Zero-Copy KV Cache Reusecited by 1
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026AIRGuard: Guarding Agent Actions with Runtime Authority Controlcited by 9
- 2026The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneckcited by 8
- 2026DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Modelcited by 3
- 2026FAVA: Formal Authorization for Verified Agents with Evidence-Backed Permission Graphscited by 2
- 2026Depth-Dependent Indirect Prompt Injection in Tool-Calling ReAct Agents: Injection Depth, Payload Framing, and Turn-Budget Sensitivitycited by 0
- 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectivescited by 0
- 2026HARDE: Optimizing Agent Harnesses for Runtime Risk Detection and Execution Controlcited by 0
- 2026HARDE: Optimizing Agent Harnesses for Runtime Risk Detection and Execution Control29 Sep 2026
- 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectives15 Sep 2026
- 2026DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Model6 Aug 2026
- 2026FAVA: Formal Authorization for Verified Agents with Evidence-Backed Permission Graphs29 Jul 2026
- 2026Depth-Dependent Indirect Prompt Injection in Tool-Calling ReAct Agents: Injection Depth, Payload Framing, and Turn-Budget Sensitivity29 May 2026
- 2026AIRGuard: Guarding Agent Actions with Runtime Authority Control27 May 2026
- 2026The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneck11 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026S3: Improving Agent Safety through Multi-Stage Defense
- 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection
- 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning
- 2025A Safety and Security Framework for Real-World Agentic Systems
- 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment
- 2026ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents
- 2025Agent Safety Alignment via Reinforcement Learning
- 2025AgentGuard: Repurposing Agentic Orchestrator for Safety Evaluation of Tool Orchestration