mlsanityData to 7 Oct 2026

ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning

arXiv 2503.22738 ↗First posted 26 Mar 2025cs.LG cs.CRcited by 144

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

Autonomous agents powered by foundation models have seen widespread adoption across various real-world applications. However, they remain highly vulnerable to malicious instructions and attacks, which can result in severe consequences such as privacy breaches and financial losses. More critically, existing guardrails for LLMs are not applicable due to the complex and dynamic nature of agents. To tackle these challenges, we propose ShieldAgent, the first guardrail agent designed to enforce explicit safety policy compliance for the action trajectory of other protected agents through logical reasoning. Specifically, ShieldAgent first constructs a safety policy model by extracting verifiable rules from policy documents and structuring them into a set of action-based probabilistic rule circuits. Given the action trajectory of the protected agent, ShieldAgent retrieves relevant rule circuits and generates a shielding plan, leveraging its comprehensive tool library and executable code for formal verification. In addition, given the lack of guardrail benchmarks for agents, we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent instructions and action trajectories, collected via SOTA attacks across 6 web environments and 7 risk categories. Experiments show that ShieldAgent achieves SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior methods by 11.3% on average with a high recall of 90.1%. Additionally, ShieldAgent reduces API queries by 64.7% and inference time by 58.2%, demonstrating its high precision and efficiency in safeguarding agents.

Key facts. Cited by 144 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 24 of its 37 references that are in the map. First posted on arXiv 26 Mar 2025.

Builds on

24 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning8 Jul
  2. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun

Other references · 22

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  2. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 402
  5. 2024Agent Workflow Memorycited by 342
  6. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  7. 2019Control Barrier Functions: Theory and Applicationscited by 332
  8. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  9. 2023A Language Agent for Autonomous Drivingcited by 161
  10. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  11. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  12. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
10 more
  1. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  2. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 112
  3. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agentscited by 87
  4. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  5. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 69
  6. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 66
  7. 2024SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanationscited by 37
  8. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  9. 2021Knowledge Enhanced Machine Learning Pipeline against Diverse Adversarial Attackscited by 24
  10. 2017Symbolic LTLf Synthesiscited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 35; 2026: 109.

144 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  3. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  4. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  5. 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 46
  6. 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 34
  7. 2025The Devil behind the mask: An emergent safety vulnerability of Diffusion LLMscited by 32
  8. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 31
  9. 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosiscited by 30
  10. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 28
  11. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  12. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
38 more
  1. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  2. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  3. 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Predictioncited by 25
  4. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  5. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  6. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 20
  7. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  8. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correctioncited by 14
  9. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  10. 2025TraceAegis: Securing LLM-Based Agents via Hierarchical and Behavioral Anomaly Detectioncited by 11
  11. 2026SafePred: A Predictive Guardrail for Computer-Using Agents via World Modelscited by 11
  12. 2026ChainCaps: Composition-Safe Tool-Using Agents via Monotonic Capability Attenuationcited by 11
  13. 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
  14. 2025Secure and Efficient Access Control for Computer-Use Agents via Context Spacecited by 10
  15. 2025Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMscited by 9
  16. 2025AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Modelscited by 9
  17. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysiscited by 9
  18. 2025Systems Security Foundations for Agentic Computingcited by 9
  19. 2026AgentGuardian: Learning Access Control Policies to Govern AI Agent Behaviorcited by 9
  20. 2026AIRGuard: Guarding Agent Actions with Runtime Authority Controlcited by 9
  21. 2025Maris: A Formally Verifiable Privacy Policy Enforcement Paradigm for Multi-Agent Collaboration Systemscited by 8
  22. 2025Exposing Weak Links in Multi-Agent Systems under Adversarial Promptingcited by 8
  23. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
  24. 2026HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmarkcited by 8
  25. 2025Graph2Eval: Automatic Multimodal Task Generation for Agents via Knowledge Graphscited by 7
  26. 2026Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Usecited by 7
  27. 2026On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignmentcited by 6
  28. 2025From EduVisBench to EduVisAgent: A Benchmark and Multi-Agent Framework for Reasoning-Driven Pedagogical Visualizationcited by 5
  29. 2025ALRPHFS: Adversarially Learned Risk Patterns with Hierarchical Fast & Slow Reasoning for Robust Agent Defensecited by 5
  30. 2026MirrorGuard: Toward Secure Computer-Use Agents via Simulation-to-Real Reasoning Correctioncited by 5
  31. 2026StepShield: When, Not Whether to Intervene on Rogue Agentscited by 5
  32. 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistantscited by 5
  33. 2026Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codexcited by 5
  34. 2026TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modelingcited by 5
  35. 2026Deontic Policies for Runtime Governance of Agentic AI Systemscited by 5
  36. 2026PolicyGuard: A Dialogue-Grounded Sub-Agent Verifier for Policy Adherence in LLM Agentscited by 5
  37. 2025SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainabilitycited by 4
  38. 2025QuadSentinel: Sequent Safety for Machine-Checkable Control in Multi-agent Systemscited by 4

50 most cited of 144 All 144 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systems
  2. 2026AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security
  3. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection
  4. 2025LlamaFirewall: An open source guardrail system for building secure AI agents
  5. 2026StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing
  6. 2026SingGuard-NSFA: Extensible Guardrails for Agentic AI via Generative Reasoning and Real-Time Classification
  7. 2025Enforcing Temporal Constraints for LLM Agents
  8. 2026ConsisGuard: Aligning Safety Deliberation with Policy Enforcement in LLM Guardrails