# RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment

arXiv: https://arxiv.org/abs/2610.09294 · HTML: https://mlsanity.com/paper/2610.09294/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 29 of its 69 references that are in the map.
- First posted on arXiv 7 Oct 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 29 of 29)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles](https://mlsanity.com/paper/2106.11810/) (2021)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents](https://mlsanity.com/paper/2502.09560/) (2025)
- [SafeAgentBench: A Benchmark for Safe Task Planning of Embodied LLM Agents](https://mlsanity.com/paper/2412.13178/) (2024)
- [VLASH: Real-Time VLAs via Future-State-Aware Asynchronous Inference](https://mlsanity.com/paper/2512.01031/) (2025)
- [A Framework for Benchmarking and Aligning Task-Planning Safety in LLM-Based Embodied Agents](https://mlsanity.com/paper/2504.14650/) (2025)
- [SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents](https://mlsanity.com/paper/2509.25885/) (2025)
- [SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning](https://mlsanity.com/paper/2605.09423/) (2026)
- [SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds](https://mlsanity.com/paper/2512.01078/) (2025)
- [TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments](https://mlsanity.com/paper/2602.02459/) (2026)
- [AsyncVLA: An Asynchronous VLA for Fast and Robust Navigation on the Edge](https://mlsanity.com/paper/2602.13476/) (2026)
- [Bench2Drive-VL: Benchmarks for Closed-Loop Autonomous Driving with Vision-Language Models](https://mlsanity.com/paper/2604.01259/) (2026)
- [FutureRTC: Real-Time Robot Execution with Anticipatory-Conditioned Action Chunking](https://mlsanity.com/paper/2607.24008/) (2026)
- [DeliveryBench: Can Agents Earn Profit in Real World?](https://mlsanity.com/paper/2512.19234/) (2025)
- [BeSafe-Bench: Unveiling Behavioral Safety Risks of Situated Agents in Functional Environments](https://mlsanity.com/paper/2603.25747/) (2026)
- [UrbanVLA: A Vision-Language-Action Model for Urban Micromobility](https://mlsanity.com/paper/2510.23576/) (2025)
- [ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models](https://mlsanity.com/paper/2606.27079/) (2026)
- [SidewalkBench: Benchmarking Visual Navigation on Urban Sidewalks](https://mlsanity.com/paper/2606.16953/) (2026)
- [Slow Brain, Fast Planner: Latency-Resilient VLM-Augmented Urban Navigation](https://mlsanity.com/paper/2606.20458/) (2026)
- [SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agents](https://mlsanity.com/paper/2607.14543/) (2026)
- [Reinforcement Learning for Real-Time Vision-Language-Action Policies](https://mlsanity.com/paper/2609.18207/) (2026)
- [DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum](https://mlsanity.com/paper/2609.19801/) (2026)
- [DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?](https://mlsanity.com/paper/2606.12402/) (2026)
- [The Speedup Paradox: Rethinking Inference Speed-Quality Trade-off in Embodied Tasks](https://mlsanity.com/paper/2606.28529/) (2026)
- [PACE: Adaptive Budget Allocation for Time-Efficient Embodied Planning](https://mlsanity.com/paper/2608.03034/) (2026)
- [ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs](https://mlsanity.com/paper/2609.10895/) (2026)

## Abstract (arXiv)

Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments. As agents move into the physical world, embodied safety becomes increasingly important: failures can cause human injury and costly hardware damage. Beyond selecting safe actions, embodied agents must also operate under real-time constraints: the physical world does not pause while an agent reasons. As pedestrians move and vehicles approach during inference, an action that appears safe at observation time may become unsafe before execution. Real-time embodied safety therefore depends on both decision quality and decision latency. We introduce RT-SAFE, a simulated urban benchmark for evaluating embodied-agent safety under real-time constraints. RT-SAFE combines navigation tasks with moving actors, environmental hazards, and traffic rules, while allowing the world to evolve throughout inference and action execution. Across eight VLMs, agents achieve high task completion yet almost never complete safely: in the hardest setting, only 0.7% of episodes finish without a safety event. More strikingly, matched static and real-time evaluations yield task completion rates of 91.3% and 94.1%, respectively, while real-time execution increases collisions by 12.3×. These results reveal that standard task success can mask substantial safety failures, and that decision latency itself can become a source of physical risk. Finally, we show that RT-SAFE can support offline RL training and substantially reduce collision rates while achieving strong task completion.