# Can AI Perceive Physical Danger and Intervene?

arXiv: https://arxiv.org/abs/2509.21651 · HTML: https://mlsanity.com/paper/2509.21651/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 26 of 26)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models](https://mlsanity.com/paper/2311.17600/) (2023)
- [An Overview of Catastrophic AI Risks](https://mlsanity.com/paper/2306.12001/) (2023)
- [Real-Time Execution of Action Chunking Flow Policies](https://mlsanity.com/paper/2506.07339/) (2025)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable](https://mlsanity.com/paper/2503.00555/) (2025)
- [A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation](https://mlsanity.com/paper/2305.11391/) (2023)
- [VLSBench: Unveiling Visual Leakage in Multimodal Safety](https://mlsanity.com/paper/2411.19939/) (2024)
- [Multimodal Situational Safety](https://mlsanity.com/paper/2410.06172/) (2024)
- [Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?](https://mlsanity.com/paper/2506.18183/) (2025)
- [Generating Robot Constitutions & Benchmarks for Semantic Safety](https://mlsanity.com/paper/2503.08663/) (2025)
- [Safety Aware Task Planning via Large Language Models in Robotics](https://mlsanity.com/paper/2503.15707/) (2025)
- [Benchmarking the Robustness of Agentic Systems to Adversarially-Induced Harms](https://mlsanity.com/paper/2508.16481/) (2025)
- [Defining and Evaluating Physical Safety for Large Language Models](https://mlsanity.com/paper/2411.02317/) (2024)

## Built on by (most cited first, 14 of 14)

- [Evaluating Gemini Robotics Policies in a Veo World Simulator](https://mlsanity.com/paper/2512.10675/) (2025)
- [EMBGuard: Constructing Hazard-Aware Guardrails for Safe Planning in Embodied Agents](https://mlsanity.com/paper/2605.30924/) (2026)
- [RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agents](https://mlsanity.com/paper/2605.19328/) (2026)
- [A Hazard-Informed Data Pipeline for Robotics Physical Safety](https://mlsanity.com/paper/2603.06130/) (2026)
- [HomeSafe-Bench: Evaluating Vision-Language Models on Unsafe Action Detection for Embodied Agents in Household Scenarios](https://mlsanity.com/paper/2603.11975/) (2026)
- [SafetyALFRED: Evaluating Safety-Conscious Planning of Multimodal Large Language Models](https://mlsanity.com/paper/2604.19638/) (2026)
- [ICAT: Incident-Case-Grounded Adaptive Testing for Physical-Risk Prediction in Embodied World Models](https://mlsanity.com/paper/2604.16405/) (2026)
- [Probing Collision Grounding in Vision-Language Models for Safe Human-Robot Collaboration](https://mlsanity.com/paper/2605.31196/) (2026)
- [VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoring](https://mlsanity.com/paper/2606.03954/) (2026)
- [Towards Trustworthy Physical Intelligence: From Theory to Practice Across Life Cycle](https://mlsanity.com/paper/2607.22877/) (2026)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [ARC: Autonomous Robotics Compliance A Three-Layer Governance Architecture for Deployed Autonomous Systems](https://mlsanity.com/paper/2609.12932/) (2026)
- [PlanGuard: A Guardrail for Multi-Step Plan Safety in Embodied Agents](https://mlsanity.com/paper/2609.32801/) (2026)
- [Visual Grounding Safety in Vision-Language Models](https://mlsanity.com/paper/2610.05637/) (2026)

## Abstract (arXiv)

When AI interacts with the physical world -- as a robot or an assistive agent -- new safety challenges emerge beyond those of purely ``digital AI". In such interactions, the potential for physical harm is direct and immediate. How well do state-of-the-art foundation models understand common-sense facts about physical safety, e.g. that a box may be too heavy to lift, or that a hot cup of coffee should not be handed to a child? In this paper, our contributions are three-fold: first, we develop a highly scalable approach to continuous physical safety benchmarking of Embodied AI systems, grounded in real-world injury narratives and operational safety constraints. To probe multi-modal safety understanding, we turn these narratives and constraints into photorealistic images and videos capturing transitions from safe to unsafe states, using advanced generative models. Secondly, we comprehensively analyze the ability of major foundation models to perceive risks, reason about safety, and trigger interventions; this yields multi-faceted insights into their deployment readiness for safety-critical agentic applications. Finally, we develop a post-training paradigm to teach models to explicitly reason about embodiment-specific safety constraints provided through system instructions. The resulting models generate thinking traces that make safety reasoning interpretable and transparent, achieving state of the art performance in constraint satisfaction evaluations. The benchmark is released at https://asimov-benchmark.github.io/v2