mlsanityData to 7 Oct 2026

GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning

arXiv 2406.09187 ↗First posted 13 Jun 2024cs.LGcited by 113

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 2 more
Abstract (arXiv)

The rapid advancement of large language model (LLM) agents has raised new concerns regarding their safety and security. In this paper, we propose GuardAgent, the first guardrail agent to protect target agents by dynamically checking whether their actions satisfy given safety guard requests. Specifically, GuardAgent first analyzes the safety guard requests to generate a task plan, and then maps this plan into guardrail code for execution. By performing the code execution, GuardAgent can deterministically follow the safety guard request and safeguard target agents. In both steps, an LLM is utilized as the reasoning component, supplemented by in-context demonstrations retrieved from a memory module storing experiences from previous tasks. In addition, we propose two novel benchmarks: EICU-AC benchmark to assess the access control for healthcare agents and Mind2Web-SC benchmark to evaluate the safety policies for web agents. We show that GuardAgent effectively moderates the violation actions for different types of agents on these two benchmarks with over 98% and 83% guardrail accuracies, respectively. Project page: https://guardagent.github.io/

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 22 of its 31 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

22 of 31 references.

References · 22

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  5. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  6. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  7. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  11. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
10 more
  1. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  3. 2024Agent Hospital: A Simulacrum of Hospital with Evolvable Medical Agentscited by 210
  4. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  5. 2023FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Designcited by 163
  6. 2023A Language Agent for Autonomous Drivingcited by 161
  7. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
  8. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Contentcited by 65
  9. 2023Personalized Autonomous Driving with Large Language Models: Field Experimentscited by 46
  10. 2024DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Drivingcited by 41

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 41; 2026: 64.

113 papers.

Sort:

By area:

  1. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  2. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  3. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  4. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 144
  5. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  6. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  7. 2025How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behaviorcited by 95
  8. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  9. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 89
  10. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  11. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  12. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
38 more
  1. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  3. 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 46
  4. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  5. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  6. 2026Towards Verifiably Safe Tool Use for LLM Agentscited by 34
  7. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  8. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  9. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  10. 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Predictioncited by 25
  11. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 25
  12. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 24
  13. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  14. 2024When LLMs Go Online: The Emerging Threat of Web-Enabled LLMscited by 22
  15. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  16. 2026Why Reasoning Fails to Plan: A Planning-Centric Analysis of Long-Horizon Decision Making in LLM Agentscited by 22
  17. 2025UDora: A Unified Red Teaming Framework against LLM Agents by Dynamically Hijacking Their Own Reasoningcited by 21
  18. 2025WebGuard: Building a Generalizable Guardrail for Web Agentscited by 20
  19. 2025Security of Internet of Agents: Attacks and Countermeasurescited by 17
  20. 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policiescited by 16
  21. 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 15
  22. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
  23. 2025Think Twice Before You Act: Enhancing Agent Behavioral Safety with Thought Correctioncited by 14
  24. 2024Swiss Cheese Model for AI Safety: A Taxonomy and Reference Architecture for Multi-Layered Guardrails of Foundation Model Based Agentscited by 13
  25. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 12
  26. 2025Who's the Mole? Modeling and Detecting Intention-Hiding Malicious Agents in LLM-Based Multi-Agent Systemscited by 12
  27. 2026The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agentscited by 12
  28. 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
  29. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
  30. 2026AIR: Improving Agent Safety through Incident Responsecited by 8
  31. 2026The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneckcited by 8
  32. 2025TAI3: Testing Agent Integrity in Interpreting User Intentcited by 7
  33. 2025SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulatorcited by 6
  34. 2025Achilles Heel of Distributed Multi-Agent Systemscited by 5
  35. 2025Enhancing LLM Agent Safety via Causal Influence Promptingcited by 5
  36. 2026StepShield: When, Not Whether to Intervene on Rogue Agentscited by 5
  37. 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistantscited by 5
  38. 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systemscited by 5

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systems
  2. 2025LlamaFirewall: An open source guardrail system for building secure AI agents
  3. 2025Enforcing Temporal Constraints for LLM Agents
  4. 2025AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents
  5. 2026Toward Safe LLM Agents: A Survey of Specification, Verification, and Enforcement
  6. 2026Autonomous LLM Agents & CTFs: A Second Look
  7. 2026ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents
  8. 2025ASTRA: Agentic Steerability and Risk Assessment Framework