# Failure Prediction with Statistical Guarantees for Vision-Based Robot Control

arXiv: https://arxiv.org/abs/2202.05894 · HTML: https://mlsanity.com/paper/2202.05894/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2022.

## Built on by (most cited first, 21 of 21)

- [SAFE: Multitask Failure Detection for Vision-Language-Action Models](https://mlsanity.com/paper/2506.09937/) (2025)
- [Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progress](https://mlsanity.com/paper/2410.04640/) (2024)
- [Failure Prediction at Runtime for Generative Robot Policies](https://mlsanity.com/paper/2510.09459/) (2025)
- [A System-Level View on Out-of-Distribution Data in Robotics](https://mlsanity.com/paper/2212.14020/) (2022)
- [CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2601.02295/) (2026)
- [Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehicles](https://mlsanity.com/paper/2207.12380/) (2022)
- [Conformal Methods for Quantifying Uncertainty in Spatiotemporal Data: A Survey](https://mlsanity.com/paper/2209.03580/) (2022)
- [Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring](https://mlsanity.com/paper/2605.30834/) (2026)
- [RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fields](https://mlsanity.com/paper/2412.02818/) (2024)
- [Conformal Policy Learning for Sensorimotor Control Under Distribution Shifts](https://mlsanity.com/paper/2311.01457/) (2023)
- [SafeMimic: Towards Safe and Autonomous Human-to-Robot Imitation for Mobile Manipulation](https://mlsanity.com/paper/2506.15847/) (2025)
- [CARE: Enhancing Safety of Visual Navigation through Collision Avoidance via Repulsive Estimation](https://mlsanity.com/paper/2506.03834/) (2025)
- [Out of Distribution Detection via Domain-Informed Gaussian Process State Space Models](https://mlsanity.com/paper/2309.06655/) (2023)
- [Decoding Task Progress from VLA Representations](https://mlsanity.com/paper/2608.13474/) (2026)
- [Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policies](https://mlsanity.com/paper/2507.06519/) (2025)
- [SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration](https://mlsanity.com/paper/2608.04246/) (2026)
- [PAC-Based Formal Verification for Out-of-Distribution Data Detection](https://mlsanity.com/paper/2304.01592/) (2023)
- [Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models](https://mlsanity.com/paper/2604.20472/) (2026)
- [VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Models](https://mlsanity.com/paper/2605.29605/) (2026)
- [FATE-VLA:Failue-aware test generation for vision-language-action models](https://mlsanity.com/paper/2606.02307/) (2026)
- [GUARD: Grounding Uncertainty and Ablation-Based Risk Detection for Diffusion-Based VLAs](https://mlsanity.com/paper/2608.04510/) (2026)

## Abstract (arXiv)

We are motivated by the problem of performing failure prediction for safety-critical robotic systems with high-dimensional sensor observations (e.g., vision). Given access to a black-box control policy (e.g., in the form of a neural network) and a dataset of training environments, we present an approach for synthesizing a failure predictor with guaranteed bounds on false-positive and false-negative errors. In order to achieve this, we utilize techniques from Probably Approximately Correct (PAC)-Bayes generalization theory. In addition, we present novel class-conditional bounds that allow us to trade-off the relative rates of false-positive vs. false-negative errors. We propose algorithms that train failure predictors (that take as input the history of sensor observations) by minimizing our theoretical error bounds. We demonstrate the resulting approach using extensive simulation and hardware experiments for vision-based navigation with a drone and grasping objects with a robotic manipulator equipped with a wrist-mounted RGB-D camera. These experiments illustrate the ability of our approach to (1) provide strong bounds on failure prediction error rates (that closely match empirical error rates), and (2) improve safety by predicting failures.