mlsanityData to 7 Oct 2026

SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents

arXiv 2412.13178 ↗First posted 17 Dec 2024cs.CR cs.AI cs.ROcited by 102

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

With the integration of large language models (LLMs), embodied agents have strong capabilities to understand and plan complicated natural language instructions. However, a foreseeable issue is that those embodied agents can also flawlessly execute some hazardous tasks, potentially causing damages in the real world. Existing benchmarks predominantly overlook critical safety risks, focusing solely on planning performance, while a few evaluate LLMs' safety awareness only on non-interactive image-text data. To address this gap, we present SafeAgentBench -- the first comprehensive benchmark for safety-aware task planning of embodied LLM agents in interactive simulation environments, covering both explicit and implicit hazards. SafeAgentBench includes: (1) an executable, diverse, and high-quality dataset of 750 tasks, rigorously curated to cover 10 potential hazards and 3 task types; (2) SafeAgentEnv, a universal embodied environment with a low-level controller, supporting multi-agent execution with 17 high-level actions for 9 state-of-the-art baselines; and (3) reliable evaluation methods from both execution and semantic perspectives. Experimental results show that, although agents based on different design frameworks exhibit substantial differences in task success rates, their overall safety awareness remains weak. The most safety-conscious baseline achieves only a 10% rejection rate for detailed hazardous tasks. Moreover, simply replacing the LLM driving the agent does not lead to notable improvements in safety awareness. Dataset and codes are available in https://github.com/shengyin1224/SafeAgentBench and https://huggingface.co/datasets/safeagentbench/SafeAgentBench.

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 32 of its 49 references that are in the map. First posted on arXiv 17 Dec 2024.

Builds on

32 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents8 Aug
  2. 2024"Don't forget to put the milk back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situations12 Apr
  3. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents18 Jan

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  10. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  11. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  12. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
17 more
  1. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  2. 2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 358
  3. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  4. 2021How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learnedcited by 201
  5. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 131
  6. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 81
  7. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  8. 2024LoTa-Bench: Benchmarking Language-oriented Task Planners for Embodied Agentscited by 51
  9. 2023Towards End-to-End Embodied Decision Making via Multi-modal Large Language Model: Explorations with GPT4-Vision and Beyondcited by 50
  10. 2024Self-Alignment of Large Language Models via Monopolylogue-based Social Scene Simulationcited by 46
  11. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41
  12. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systemscited by 38
  13. 2024HAZARD Challenge: Embodied Decision Making in Dynamically Changing Environmentscited by 32
  14. 2024Exploring the Robustness of Decision-Level Through Adversarial Attacks on LLM-Based Embodied Modelscited by 31
  15. 2024Physical Backdoor Attack can Jeopardize Driving with Vision-Large-Language Modelscited by 30
  16. 2024MLDT: Multi-Level Decomposition for Complex Long-Horizon Robotic Task Planning with Open-Source Large Language Modelcited by 15
  17. 2024Multi-Agent Planning Using Visual Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 33; 2026: 68.

102 papers.

Sort:

By area:

  1. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  2. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  4. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 71
  5. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  6. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  7. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  8. 2025Building a Foundational Guardrail for General Agentic Systems via Synthetic Datacited by 26
  9. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  10. 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Predictioncited by 25
  11. 2025Safety Aware Task Planning via Large Language Models in Roboticscited by 24
  12. 2025Subtle Risks, Critical Failures: A Framework for Diagnosing Physical Safety of LLMs for Embodied Decision Makingcited by 19
38 more
  1. 2026Self-Improvements in Modern Agentic Systems: A Surveycited by 17
  2. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 15
  3. 2026ClawSafety: "Safe" LLMs, Unsafe Agentscited by 15
  4. 2025SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agentscited by 14
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  6. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agentscited by 13
  7. 2025Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderationcited by 12
  8. 2026On Safety Risks in Experience-Driven Self-Evolving Agentscited by 12
  9. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
  10. 2025RoboSafe: Safeguarding Embodied Agents via Executable Safety Logiccited by 11
  11. 2025Is Safety Standard Same for Everyone? User-Specific Safety Evaluation of Large Language Modelscited by 10
  12. 2025Agent Safety Alignment via Reinforcement Learningcited by 10
  13. 2025Safety of Embodied Navigation: A Surveycited by 9
  14. 2025Measuring Harmfulness of Computer-Using Agentscited by 8
  15. 2026AIR: Improving Agent Safety through Incident Responsecited by 8
  16. 2026VIGIL: Runtime Enforcement of Behavioral Specifications in AI Agent Skillscited by 8
  17. 2025REI-Bench: Can Embodied Agents Understand Vague Human Instructions in Task Planning?cited by 7
  18. 2026Learning When to Act or Refuse: Guarding Agentic Reasoning Models for Safe Multi-Step Tool Usecited by 7
  19. 2025PLANET: A Collection of Benchmarks for Evaluating LLMs' Planning Capabilitiescited by 6
  20. 2025SafeAgent: Safeguarding LLM Agents via an Automated Risk Simulatorcited by 6
  21. 2026HazardArena: Evaluating Semantic Safety in Vision-Language-Action Modelscited by 6
  22. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Datacited by 5
  23. 2026StepShield: When, Not Whether to Intervene on Rogue Agentscited by 5
  24. 2026SafeClaw-R: Towards Safe and Secure Multi-Agent Personal Assistantscited by 5
  25. 2026Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluationcited by 5
  26. 2025HomeSafeBench: Benchmarking Embodied Vision-Language Models in Free-Exploration Home Safety Inspectioncited by 4
  27. 2025SafeSearch: Automated Red-Teaming of LLM-Based Search Agentscited by 4
  28. 2025MADRA: Multi-Agent Debate for Risk-Aware Embodied Planningcited by 4
  29. 2026SAVeS: Steering Safety Judgments in Vision-Language Models via Semantic Cuescited by 4
  30. 2026Using large language models for embodied planning introduces systematic safety riskscited by 4
  31. 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agentscited by 4
  32. 2025Are All Prompt Components Value-Neutral? Understanding the Heterogeneous Adversarial Robustness of Dissected Prompt in Large Language Modelscited by 3
  33. 2025Embodied AI: Emerging Risks and Opportunities for Policy Actioncited by 3
  34. 2026ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AIcited by 3
  35. 2026LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratoriescited by 3
  36. 2026HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Taskcited by 3
  37. 2026BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environmentscited by 3
  38. 2026AgentLens: Interpretable Safety Steering via Mechanistic Subspaces for Multi-Turn Coding Agentcited by 3

50 most cited of 102 All 102 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agents
  2. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Tasks
  3. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment
  4. 2026RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment
  5. 2026From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agents
  6. 2026ForesightSafety-SAGE:A Fully Automated Scenario Generation and Safety Evaluation Framework for LLM Agents
  7. 2026Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses
  8. 2026SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfaces