# Foundational World Models Accurately Detect Bimanual Manipulator Failures

arXiv: https://arxiv.org/abs/2603.06987 · HTML: https://mlsanity.com/paper/2603.06987/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 7 Mar 2026.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 25 of 25)

- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Do Deep Generative Models Know What They Don't Know?](https://mlsanity.com/paper/1810.09136/) (2018)
- [CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances](https://mlsanity.com/paper/2007.08176/) (2020)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Why Normalizing Flows Fail to Detect Out-of-Distribution Data](https://mlsanity.com/paper/2006.08545/) (2020)
- [Classification-Based Anomaly Detection for General Data](https://mlsanity.com/paper/2005.02359/) (2020)
- [Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder](https://mlsanity.com/paper/2003.02977/) (2020)
- [C2AE: Class Conditioned Auto-Encoder for Open-set Recognition](https://mlsanity.com/paper/1904.01198/) (2019)
- [Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers](https://mlsanity.com/paper/1809.03576/) (2018)
- [Semantic Anomaly Detection with Large Language Models](https://mlsanity.com/paper/2305.11307/) (2023)
- [From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignment](https://mlsanity.com/paper/2502.01828/) (2025)
- [Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policies](https://mlsanity.com/paper/2503.08558/) (2025)
- [Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis](https://mlsanity.com/paper/2502.00935/) (2025)
- [Multi-Task Interactive Robot Fleet Learning with Visual World Models](https://mlsanity.com/paper/2410.22689/) (2024)
- [Uncertainty-aware Latent Safety Filters for Avoiding Out-of-Distribution Failures](https://mlsanity.com/paper/2505.00779/) (2025)
- [Further Analysis of Outlier Detection with Deep Generative Models](https://mlsanity.com/paper/2010.13064/) (2020)
- [Improving Image Autoencoder Embeddings with Perceptual Loss](https://mlsanity.com/paper/2001.03444/) (2020)
- [Safe, Out-of-Distribution-Adaptive MPC with Conformalized Neural Network Ensembles](https://mlsanity.com/paper/2406.02436/) (2024)
- [World Model Failure Classification and Anomaly Detection for Autonomous Inspection](https://mlsanity.com/paper/2602.16182/) (2026)
- [Optimal Control of Mechanical Ventilators with Learned Respiratory Dynamics](https://mlsanity.com/paper/2411.07971/) (2024)
- [Towards a Framework for Comparing the Complexity of Robotic Tasks](https://mlsanity.com/paper/2202.09892/) (2022)
- [Improving the Resilience of Quadrotors in Underground Environments by Combining Learning-based and Safety Controllers](https://mlsanity.com/paper/2509.02808/) (2025)

## Built on by (most cited first, 11 of 11)

- [Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latents](https://mlsanity.com/paper/2606.23085/) (2026)
- [Failure Detection for Surgical Robot Imitation Policies via Flow-Matching World Modeling](https://mlsanity.com/paper/2607.27511/) (2026)
- [Robots Need More than VLA and World Models](https://mlsanity.com/paper/2606.06556/) (2026)
- [World Models for Embodied Intelligence: From Plausible to Controllable to Actionable](https://mlsanity.com/paper/2609.16697/) (2026)
- [Where World Models Break: Natural-Input Failure Discovery](https://mlsanity.com/paper/2608.22421/) (2026)
- [GPU-Accelerated Astrodynamics World Models for Spacecraft Rendezvous and Proximity Operations](https://mlsanity.com/paper/2609.03067/) (2026)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [FARM: Reading Failure Signals from the Internal Predictive States of a Frozen Robotic World Model](https://mlsanity.com/paper/2609.11445/) (2026)
- [Behavioral Monitoring of JEPA World Models with Jacobian Centroids](https://mlsanity.com/paper/2609.33940/) (2026)
- [When World Models Lie: Adaptive Safety Analysis Under Wrong Imaginations](https://mlsanity.com/paper/2609.34300/) (2026)
- [Stochastic World Models for Verifying Vision-Based Neural Feedback Systems](https://mlsanity.com/paper/2609.38120/) (2026)

## Abstract (arXiv)

Deploying visuomotor robots at scale is challenging due to the potential for anomalous failures to degrade performance, cause damage, or endanger human life. Bimanual manipulators are no exception; these robots have vast state spaces comprised of high-dimensional images and proprioceptive signals. Explicitly defining failure modes within such state spaces is infeasible. In this work, we overcome these challenges by training a probabilistic, history informed, world model within the compressed latent space of a pretrained vision foundation model (NVIDIA's Cosmos Tokenizer). The model outputs uncertainty estimates alongside its predictions that serve as non-conformity scores within a conformal prediction framework. We use these scores to develop a runtime monitor, correlating periods of high uncertainty with anomalous failures. To test these methods, we use the simulated Push-T environment and the Bimanual Cable Manipulation dataset, the latter of which we introduce in this work. This new dataset features trajectories with multiple synchronized camera views, proprioceptive signals, and annotated failures from a challenging data center maintenance task. We benchmark our methods against baselines from the anomaly detection and out-of-distribution detection literature, and show that our approach considerably outperforms statistical techniques. Furthermore, we show that our approach requires approximately one twentieth of the trainable parameters as the next-best learning-based approach, yet outperforms it by 3.8% in terms of failure detection rate, paving the way toward safely deploying manipulator robots in real-world environments where reliability is non-negotiable.