# AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection

arXiv: https://arxiv.org/abs/2502.11448 · HTML: https://mlsanity.com/paper/2502.11448/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 17 Feb 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 27 of 27)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2404.03027/) (2024)
- [Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agents](https://mlsanity.com/paper/2405.02957/) (2024)
- [Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents](https://mlsanity.com/paper/2411.06559/) (2024)
- [The Impact of Reasoning Step Length on Large Language Models](https://mlsanity.com/paper/2401.04925/) (2024)
- [EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage](https://mlsanity.com/paper/2409.11295/) (2024)
- [GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning](https://mlsanity.com/paper/2406.09187/) (2024)
- [Conversational Health Agents: A Personalized LLM-Powered Agent Framework](https://mlsanity.com/paper/2310.02374/) (2023)
- [AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies](https://mlsanity.com/paper/2407.17436/) (2024)
- [RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content](https://mlsanity.com/paper/2403.13031/) (2024)
- [TrustAgent: Towards Safe and Trustworthy LLM-based Agents](https://mlsanity.com/paper/2402.01586/) (2024)
- [Testing Language Model Agents Safely in the Wild](https://mlsanity.com/paper/2311.10538/) (2023)
- [InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Models](https://mlsanity.com/paper/2410.22770/) (2024)
- [Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments](https://mlsanity.com/paper/2402.14672/) (2024)
- [PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents](https://mlsanity.com/paper/2402.12326/) (2024)
- [BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis](https://mlsanity.com/paper/2404.15532/) (2024)

## Built on by (most cited first, 30 of 89)

- [The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections](https://mlsanity.com/paper/2510.09023/) (2025)
- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback](https://mlsanity.com/paper/2601.10156/) (2026)
- [VeriGuard: Enhancing LLM Agent Safety via Verified Code Generation](https://mlsanity.com/paper/2510.05156/) (2025)
- [RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments](https://mlsanity.com/paper/2505.21936/) (2025)
- [AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection](https://mlsanity.com/paper/2508.01249/) (2025)
- [The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis](https://mlsanity.com/paper/2602.10453/) (2026)
- [Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents](https://mlsanity.com/paper/2602.13379/) (2026)
- [Building a Foundational Guardrail for General Agentic Systems via Synthetic Data](https://mlsanity.com/paper/2510.09781/) (2025)
- [Taming OpenClaw: Security Analysis and Mitigation of Autonomous LLM Agent Threats](https://mlsanity.com/paper/2603.11619/) (2026)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey](https://mlsanity.com/paper/2603.11088/) (2026)
- [A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?](https://mlsanity.com/paper/2505.10924/) (2025)
- [UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoning](https://mlsanity.com/paper/2503.01908/) (2025)
- [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025)
- [When Safe Skills Collide: Measuring Compositional Risk in Agent Skill Ecosystems](https://mlsanity.com/paper/2606.00448/) (2026)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Auditable Agents](https://mlsanity.com/paper/2604.05485/) (2026)
- [Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Models](https://mlsanity.com/paper/2504.19373/) (2025)
- [From Spark to Fire: Modeling and Mitigating Error Cascades in LLM-Based Multi-Agent Collaboration](https://mlsanity.com/paper/2603.04474/) (2026)
- [PSG-Agent: Personality-Aware Safety Guardrail for LLM-based Agents](https://mlsanity.com/paper/2509.23614/) (2025)
- [No Free Lunch with Guardrails](https://mlsanity.com/paper/2504.00441/) (2025)
- [AgentGuardian: Learning Access Control Policies to Govern AI Agent Behavior](https://mlsanity.com/paper/2601.10440/) (2026)
- [OpenClaw PRISM: A Zero-Fork, Defense-in-Depth Runtime Security Layer for Tool-Augmented LLM Agents](https://mlsanity.com/paper/2603.11853/) (2026)
- [Reframing LLM Agent Security as an Agent-Human Interaction Problem](https://mlsanity.com/paper/2605.24309/) (2026)
- [When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents](https://mlsanity.com/paper/2602.08995/) (2026)
- [TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling](https://mlsanity.com/paper/2605.27690/) (2026)
- [AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security](https://mlsanity.com/paper/2605.29801/) (2026)
- [Deep Research with Open-Domain Evaluation and Multi-Stage Guardrails for Safety](https://mlsanity.com/paper/2510.10994/) (2025)
- [When Bots Take the Bait: Exposing and Mitigating the Emerging Social Engineering Attack in Web Automation Agent](https://mlsanity.com/paper/2601.07263/) (2026)

## Abstract (arXiv)

The rapid advancements in Large Language Models (LLMs) have enabled their deployment as autonomous agents for handling complex tasks in dynamic environments. These LLMs demonstrate strong problem-solving capabilities and adaptability to multifaceted scenarios. However, their use as agents also introduces significant risks, including task-specific risks, which are identified by the agent administrator based on the specific task requirements and constraints, and systemic risks, which stem from vulnerabilities in their design or interactions, potentially compromising confidentiality, integrity, or availability (CIA) of information and triggering security risks. Existing defense agencies fail to adaptively and effectively mitigate these risks. In this paper, we propose AGrail, a lifelong agent guardrail to enhance LLM agent safety, which features adaptive safety check generation, effective safety check optimization, and tool compatibility and flexibility. Extensive experiments demonstrate that AGrail not only achieves strong performance against task-specific and system risks but also exhibits transferability across different LLM agents' tasks.