mlsanityData to 7 Oct 2026

AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection

arXiv 2502.11448 ↗First posted 17 Feb 2025cs.AIcited by 82

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

The rapid advancements in Large Language Models (LLMs) have enabled their deployment as autonomous agents for handling complex tasks in dynamic environments. These LLMs demonstrate strong problem-solving capabilities and adaptability to multifaceted scenarios. However, their use as agents also introduces significant risks, including task-specific risks, which are identified by the agent administrator based on the specific task requirements and constraints, and systemic risks, which stem from vulnerabilities in their design or interactions, potentially compromising confidentiality, integrity, or availability (CIA) of information and triggering security risks. Existing defense agencies fail to adaptively and effectively mitigate these risks. In this paper, we propose AGrail, a lifelong agent guardrail to enhance LLM agent safety, which features adaptive safety check generation, effective safety check optimization, and tool compatibility and flexibility. Extensive experiments demonstrate that AGrail not only achieves strong performance against task-specific and system risks but also exhibits transferability across different LLM agents' tasks.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 27 of its 51 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

27 of 51 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents19 Dec
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  3. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage17 Sep
  4. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
  5. 2024TrustAgent: Towards Safe and Trustworthy LLM-based Agents2 Feb
  6. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  7. 2023Testing Language Model Agents Safely in the Wild17 Nov
  8. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 19

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  6. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  7. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  8. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  9. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 210
  10. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attackscited by 208
  11. 2024The Impact of Reasoning Step Length on Large Language Modelscited by 152
  12. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
7 more
  1. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
  2. 2024AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policiescited by 69
  3. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  4. 2024Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environmentscited by 39
  5. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  6. 2024PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agentscited by 28
  7. 2024BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysiscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 20; 2026: 62.

82 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  3. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 64
  4. 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 46
  5. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  6. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  7. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  8. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  9. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  10. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  11. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 25
  12. 2026Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threatscited by 25
38 more
  1. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  2. 2025UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoningcited by 21
  3. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  4. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  6. 2026Auditable Agentscited by 14
  7. 2026When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystemscited by 14
  8. 2026From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaborationcited by 13
  9. 2025PSG-Agent: Personality-Aware Safety Guardrail for LLM-based Agentscited by 12
  10. 2025No Free Lunch with Guardrailscited by 10
  11. 2026OpenClaw PRISM: A Zero-Fork, Defense-in-Depth Runtime Security Layer for Tool-Augmented LLM Agentscited by 10
  12. 2026AgentGuardian: Learning Access Control Policies to Govern AI Agent Behaviorcited by 9
  13. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
  14. 2026Reframing LLM Agent Security as an Agent-Human Interaction Problemcited by 8
  15. 2026AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Securitycited by 7
  16. 2025Deep Research with Open-Domain Evaluation and Multi-Stage Guardrails for Safetycited by 6
  17. 2026When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agentcited by 6
  18. 2025ALRPHFS: Adversarially Learned Risk Patterns with Hierarchical Fast & Slow Reasoning for Robust Agent Defensecited by 5
  19. 2026ACIArena: Toward Unified Evaluation for Agent Cascading Injectioncited by 5
  20. 2026TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modelingcited by 5
  21. 2026From Risk Classification to Action Plan Remediation: A Guardrail Feedback Driven Framework for LLM Agentscited by 5
  22. 2025Your Harness is Not Secure: Benchmarking Real-world Threat of Command Line Interface Agentcited by 4
  23. 2026SpecOps: A Fully Automated AI Agent Testing Framework in Real-World GUI Environmentscited by 4
  24. 2026FragFuse: Bypassing Access Control of Large Language Model Agents via Memory-Based Query Fragmentation and Fusioncited by 4
  25. 2026Safeguarding LLM Agents from Misalignment through Provenance Analysiscited by 4
  26. 2025Don't Trust Your Upstream: Exploiting LLM Multi-Agent System via Topology-Guided Adversarial Propagationcited by 3
  27. 2026Whispers of Wealth: Red-Teaming Google's Agent Payments Protocol via Prompt Injectioncited by 3
  28. 2026SafeHarbor: Defining Precise Decision Boundaries via Hierarchical Memory-Augmented Guardrail for LLM Agent Safetycited by 3
  29. 2026SafeMCP: Proactive Power Regulation for LLM Agent Defense via Environment-Grounded Look-Ahead Reasoningcited by 3
  30. 2026TRACE: Trajectory Reasoning through Adaptive Cross-Step Evidence Aggregation for LLM Agentscited by 3
  31. 2026PROJECTMEM: A Local-First, Event-Sourced Memory and Judgment Layer for AI Coding Agentscited by 3
  32. 2026AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agentcited by 3
  33. 2026Preventing Premature Commitment in Coding Agents with an Evidence-Conditioned Execution Layercited by 3
  34. 2026DreamGuard: Efficient Runtime Guardrail for LLM Agents via Risk-Aware World Modelcited by 3
  35. 2025Learning Efficient Guardrails for Compliancecited by 2
  36. 2026MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubricscited by 2
  37. 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safetycited by 2
  38. 2026ADR: An Agentic Detection System for Enterprise Agentic AI Securitycited by 2

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systems
  2. 2025ASTRA: Agentic Steerability and Risk Assessment Framework
  3. 2025LlamaFirewall: An open source guardrail system for building secure AI agents
  4. 2026ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents
  5. 2026Toward Secure AI-Powered Penetration Testing Agents: Security Threats, Guardrails, and Architectural Perspectives
  6. 2026S3: Improving Agent Safety through Multi-Stage Defense
  7. 2026SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment
  8. 2026AIR: Improving Agent Safety through Incident Response