# Silent Failures in Physical AI: A Literature Review of Runtime Action Authorization for Autonomous Systems

arXiv: https://arxiv.org/abs/2606.00090 · HTML: https://mlsanity.com/paper/2606.00090/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in World models for navigation, one of 11 directions in World models.
- Builds on 24 of its 118 references that are in the map.
- First posted on arXiv 23 May 2026.

## Placed in

- World models / [World models for navigation](https://mlsanity.com/wm/world-models-for-navigation/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 24 of 24)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles](https://mlsanity.com/paper/1705.05065/) (2017)
- [ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills](https://mlsanity.com/paper/2302.04659/) (2023)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [LeRobot: An Open-Source Library for End-to-End Robot Learning](https://mlsanity.com/paper/2602.22818/) (2026)
- [AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security](https://mlsanity.com/paper/2601.18491/) (2026)
- [WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL](https://mlsanity.com/paper/2602.13977/) (2026)
- [VLAW: Iterative Co-Improvement of Vision-Language-Action Policy and World Model](https://mlsanity.com/paper/2602.12063/) (2026)
- [Safety Guardrails for LLM-Enabled Robots](https://mlsanity.com/paper/2503.07885/) (2025)
- [ABot-PhysWorld: Interactive World Foundation Model for Robotic Manipulation with Physics Alignment](https://mlsanity.com/paper/2603.23376/) (2026)
- [Scaling World Model for Hierarchical Manipulation Policies](https://mlsanity.com/paper/2602.10983/) (2026)
- [EvoVLA: Self-Evolving Vision-Language-Action Model](https://mlsanity.com/paper/2511.16166/) (2025)
- [Unifying Language-Action Understanding and Generation for Autonomous Driving](https://mlsanity.com/paper/2603.01441/) (2026)
- [Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines](https://mlsanity.com/paper/2604.23001/) (2026)
- [Modular Safety Guardrails Are Necessary for Foundation-Model-Enabled Robots in the Real World](https://mlsanity.com/paper/2602.04056/) (2026)
- [StyleVLA: Driving Style-Aware Vision Language Action Model for Autonomous Driving](https://mlsanity.com/paper/2603.09482/) (2026)
- [Action Hallucination in Generative Vision-Language-Action Models](https://mlsanity.com/paper/2602.06339/) (2026)
- [Recursive Belief Vision Language Action Models](https://mlsanity.com/paper/2602.20659/) (2026)
- [Judge, Then Drive: A Critic-Centric Vision Language Action Framework for Autonomous Driving](https://mlsanity.com/paper/2604.27366/) (2026)
- [From Particles to Agents: Hallucination as a Metric for Cognitive Friction in Spatial Simulation](https://mlsanity.com/paper/2601.21977/) (2026)

## Abstract (arXiv)

Physical AI systems increasingly map multimodal observations, language instructions, and learned world representations into physically consequential actions. Robotics foundation models, vision-language-action models, and world-model-based autonomous systems can condition decisions that move vehicles, robots, drones, and industrial machines. This transition exposes a safety problem that is not fully captured by conventional AI content moderation or by classical robot safety alone: a black-box model may issue a physically consequential action while appearing confident, plausible, and semantically aligned. The resulting failure can be silent, arising from sensor drift, occlusion, state-estimation error, distribution shift, hallucinated affordances, or invalid physical assumptions before downstream hardware controllers detect a violation. Across embodied foundation models, world models, robotics simulation, embodied safety benchmarks, safe control, runtime assurance, uncertainty estimation, verification, and guardrail evaluation, model capability and safety mechanisms have advanced along largely separate technical tracks. A recurring gap synthesized here is that no single stream surveyed in this review supplies a complete runtime authorization boundary between black-box Physical AI models and physical execution. The resulting analysis develops a bounded problem formulation, a definition of silent physical-action failure, a taxonomy of runtime guardrail functions, and evaluation requirements for comparing guardrails as Physical AI assurance mechanisms.