mlsanityData to 7 Oct 2026

A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agents

arXiv 2504.14650 ↗First posted 20 Apr 2025cs.AIcited by 27

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large Language Models (LLMs) exhibit substantial promise in enhancing task-planning capabilities within embodied agents due to their advanced reasoning and comprehension. However, the systemic safety of these agents remains an underexplored frontier. In this study, we present Safe-BeAl, an integrated framework for the measurement (SafePlan-Bench) and alignment (Safe-Align) of LLM-based embodied agents' behaviors. SafePlan-Bench establishes a comprehensive benchmark for evaluating task-planning safety, encompassing 2,027 daily tasks and corresponding environments distributed across 8 distinct hazard categories (e.g., Fire Hazard). Our empirical analysis reveals that even in the absence of adversarial inputs or malicious intent, LLM-based agents can exhibit unsafe behaviors. To mitigate these hazards, we propose Safe-Align, a method designed to integrate physical-world safety knowledge into LLM-based embodied agents while maintaining task-specific performance. Experiments across a variety of settings demonstrate that Safe-BeAl provides comprehensive safety validation, improving safety by 8.55 - 15.22%, compared to embodied agents based on GPT-4, while ensuring successful task completion.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 23 of its 36 references that are in the map. First posted on arXiv 20 Apr 2025.

Builds on

23 of 36 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  11. 2018VirtualHome: Simulating Household Activities via Programscited by 411
  12. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
11 more
  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  4. 2023Large Language Model Alignment: A Surveycited by 241
  5. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  6. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  7. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
  8. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  9. 2024The Vulnerability of Language Model Benchmarks: Do They Accurately Reflect True LLM Performance?cited by 22
  10. 2024Hazards in Daily Life? Enabling Robots to Proactively Detect and Resolve Anomaliescited by 19
  11. 2024LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robotscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 20.

27 papers.

Sort:

By area:

  1. 2025IS-Bench: Evaluating Interactive Safety of VLM-Driven Embodied Agents in Daily Household Taskscited by 40
  2. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  3. 2025SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agentscited by 13
  4. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  5. 2025HomeSafeBench: Benchmarking Embodied Vision-Language Models in Free-Exploration Home Safety Inspectioncited by 4
  6. 2025MADRA: Multi-Agent Debate for Risk-Aware Embodied Planningcited by 4
  7. 2026Using large language models for embodied planning introduces systematic safety riskscited by 4
  8. 2026ForesightSafety Bench: A Frontier Risk Evaluation and Governance Framework towards Safe AIcited by 3
  9. 2026LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratoriescited by 3
  10. 2026HomeGuard: VLM-based Embodied Safeguard for Identifying Contextual Risk in Household Taskcited by 3
  11. 2026LabGuard: Grounding Natural-Language Laboratory Rules into Runtime Guards for Embodied Laboratory Agentscited by 2
  12. 2026Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluationcited by 2
15 more
  1. 2025S3LoRA: Safe Spectral Sharpness-Guided Pruning in Adaptation of Agent Plannercited by 1
  2. 2026HomeSafe-Bench: Evaluating Vision-Language Models on Unsafe Action Detection for Embodied Agents in Household Scenarioscited by 1
  3. 2026EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systemscited by 1
  4. 2026SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agentscited by 1
  5. 2026Before We Trust Them: Decision-Making Failures in Navigation of Foundation Modelscited by 0
  6. 2026TSHA: A Benchmark for Visual Language Models in Trustworthy Safety Hazard Assessment Scenarioscited by 0
  7. 2026SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Modelscited by 0
  8. 2026NEXUS: Continual Learning of Symbolic Constraints for Safe and Robust Embodied Planningcited by 0
  9. 2026EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guardscited by 0
  10. 2026When Words Are Safe But Actions Kill: Probing Physical Jailbreak Beyond Textual Jailbreak in Hidden-State Risk Spacecited by 0
  11. 2026GuardianBench: A Same-Scene Instruction-Contrastive Benchmark for Latent Contextual Risk in Embodied AIcited by 0
  12. 2026Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Modelscited by 0
  13. 2026Safe Task Planning with Long-Term Graph Memory for Embodied Agentscited by 0
  14. 2026Security and Privacy in Large-Model-Driven Embodied Agents: Attacks, Defenses, and Future Directionscited by 0
  15. 2026PlanGuard: A Guardrail for Multi-Step Plan Safety in Embodied Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
  3. 2025SafePlan: Leveraging Formal Logic and Chain-of-Thought Reasoning for Enhanced Safety in LLM-based Robotic Task Planning
  4. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models
  5. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment
  6. 2026SafeBranch: Branch-Pair Safety Alignment for Embodied Agents
  7. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agents
  8. 2026AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents