mlsanityData to 14 Oct 2026

Agent-SafetyBench: Evaluating the Safety of LLM Agents

arXiv 2412.14470 ↗First posted 19 Dec 2024cs.CLcited by 306

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

As large language models (LLMs) are increasingly deployed as agents, their integration into interactive environments and tool use introduce new safety challenges beyond those associated with the models themselves. However, the absence of comprehensive benchmarks for evaluating agent safety presents a significant barrier to effective assessment and further improvement. In this paper, we introduce Agent-SafetyBench, a comprehensive benchmark designed to evaluate the safety of LLM agents. Agent-SafetyBench encompasses 349 interaction environments and 2,000 test cases, evaluating 8 categories of safety risks and covering 10 common failure modes frequently encountered in unsafe interactions. Our evaluation of 16 popular LLM agents reveals a concerning result: none of the agents achieves a safety score above 60%. This highlights significant safety challenges in LLM agents and underscores the considerable need for improvement. Through failure mode and helpfulness analysis, we summarize two fundamental safety defects in current LLM agents: lack of robustness and lack of risk awareness. Furthermore, our findings suggest that reliance on defense prompts alone may be insufficient to address these safety issues, emphasizing the need for more advanced and robust strategies. To drive progress in this area, Agent-SafetyBench has been released at https://github.com/thu-coai/Agent-SafetyBench/ to facilitate further research in agent safety evaluation and improvement.

Key facts. Cited by 306 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 23 of its 28 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

23 of 28 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents17 Dec
  2. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents19 Jun
  3. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
  4. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  5. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan
  6. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandbox25 Sep

Other references · 17

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  6. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  7. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
  8. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  9. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  10. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 85
  11. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 82
  12. 2024ShieldLM: Empowering LLMs as Aligned, Customizable and Explainable Safety Detectorscited by 51
5 more
  1. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 51
  2. 2024HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactionscited by 44
  3. 2023Flames: Benchmarking Value Alignment of LLMs in Chinesecited by 37
  4. 2023Ethicist: Targeted Training Data Extraction Through Loss Smoothed Soft Prompting and Calibrated Confidence Estimationcited by 36
  5. 2023FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicitycited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 82; 2026: 224.

306 papers.

Sort:

By area:

  1. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  2. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 152
  3. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  5. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
  6. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 89
  7. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  8. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedbackcited by 72
  9. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  10. 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 64
  11. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 62
  12. 2026SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Taskscited by 55
38 more
  1. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  2. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
  3. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  4. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agentscited by 42
  5. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 40
  6. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 39
  7. 2026WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluationcited by 37
  8. 2026ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosiscited by 36
  9. 2025RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agentscited by 35
  10. 2025Towards Trustworthy GUI Agents: A Surveycited by 34
  11. 2026ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspacescited by 33
  12. 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agentscited by 32
  13. 2025LLM Agents Should Employ Security Principlescited by 29
  14. 2025A Survey on Autonomy-Induced Security Risks in Large Model-Based Agentscited by 29
  15. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 29
  16. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  17. 2025A Novel Evaluation Benchmark for Medical LLMs: Illuminating Safety and Effectiveness in Clinical Domainscited by 28
  18. 2025AI Agentic Programming: A Survey of Techniques, Challenges, and Opportunitiescited by 28
  19. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 28
  20. 2025Nuclear Deployed: Analyzing Catastrophic Risks in Decision-making of Autonomous LLM Agentscited by 27
  21. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  22. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injectioncited by 27
  23. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  24. 2025Attractive Metadata Attack: Inducing LLM Agents to Invoke Malicious Toolscited by 24
  25. 2025A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?cited by 23
  26. 2025MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environmentscited by 23
  27. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Modelscited by 21
  28. 2025Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Makingcited by 20
  29. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  30. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluationcited by 20
  31. 2026ClawKeeper: Comprehensive Safety Protection for OpenClaw Agents Through Skills, Plugins, and Watcherscited by 20
  32. 2026Breaking the Protocol: Security Analysis of the Model Context Protocol Specification and Prompt Injection Vulnerabilities in Tool-Integrated LLM Agentscited by 19
  33. 2026From Storage to Steering: Memory Control Flow Attacks on LLM Agentscited by 19
  34. 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agentscited by 18
  35. 2025RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environmentscited by 17
  36. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 17
  37. 2026AgentAbstain: Do LLM Agents Know When Not to Act?cited by 17
  38. 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approachcited by 16

50 most cited of 306 All 306 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking
  2. 2025OffTopicEval: When Large Language Models Enter the Wrong Chat, Almost Always!
  3. 2025Efficient LLM Safety Evaluation through Multi-Agent Debate
  4. 2024Large Language Model Safety: A Holistic Survey
  5. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  6. 2023AgentBench: Evaluating LLMs as Agents
  7. 2025AgenticEval: Toward Agentic and Self-Evolving Safety Evaluation of Large Language Models
  8. 2025Check Yourself Before You Wreck Yourself: Selectively Quitting Improves LLM Agent Safety