mlsanityData to 7 Oct 2026

SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration

arXiv 2608.04246 ↗First posted 4 Aug 2026cs.RO cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

Vision-language-action policies often fail under deployment-time distribution shifts such as clutter, distractor objects, lighting changes, novel objects, altered initial states, and reworded instructions. Hidden-state-based risk probes combined with functional conformal prediction can detect rollout failures, but their reliability depends on calibration data matching deployment conditions. We introduce SAFECAST, which leverages contrast set perturbations to improve hidden-state probe training and calibration for deployment time shift. SAFECAST statistically significantly improves failure detection ROC-AUC scores over a state of the art baseline in both real-world DROID and LIBERO simulation experiments across multiple VLM backbones. We further find that SAFECAST benefits most when both visual and language contrast set perturbations are used to augment data, and that with contrast set perturbations, sim-to-real calibration leads to better probes than using real rollout data only.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 41 of its 46 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

41 of 46 references.

References · 41

  1. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  3. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  6. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  7. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  10. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  11. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  12. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
29 more
  1. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 278
  2. 2023Vision-Language Models as Success Detectorscited by 138
  3. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  4. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
  5. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  6. 2023Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulationcited by 78
  7. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 74
  8. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  9. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  10. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulationcited by 59
  11. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  12. 2025Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policiescited by 49
  13. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
  14. 2022A System-Level View on Out-of-Distribution Data in Roboticscited by 33
  15. 2024Multi-Task Interactive Robot Fleet Learning with Visual World Modelscited by 33
  16. 2021Run-Time Monitoring of Machine Learning for Robotic Perception: A Survey of Emerging Trendscited by 26
  17. 2024Robot Learning as an Empirical Science: Best Practices for Policy Evaluationcited by 22
  18. 2022Failure Prediction with Statistical Guarantees for Vision-Based Robot Controlcited by 21
  19. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  20. 2022When to Ask for Help: Proactive Interventions in Autonomous Reinforcement Learningcited by 20
  21. 2021The Importance of Being a Band: Finite-Sample Exact Distribution-Free Prediction Sets for Functional Datacited by 16
  22. 2023Asking for Help: Failure Prediction in Behavioral Cloning through Value Approximationcited by 15
  23. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  24. 2023Closing the Loop on Runtime Monitors with Fallback-Safe MPCcited by 14
  25. 2025Predictive Red Teaming: Breaking Policies Without Breaking Robotscited by 14
  26. 2020Fighting Failures with FIRE: Failure Identification to Reduce Expert Burden in Intervention-Based Learningcited by 8
  27. 2024Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Taskscited by 8
  28. 2024Contrast Sets for Evaluating Language-Guided Robot Policiescited by 7
  29. 2025Efficient Evaluation of Multi-Task Robot Policies With Active Experiment Selectioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026FailBench: How Reliable are VLMs at Judging Robot Task Success?cited by 0
  2. 2026VLA-Scope: Shift-Aware Failure Prediction for Vision-Language-Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Tri-Info: Generalizable, Interpretable Failure Prediction for VLA Models via Information Theory
  2. 2026VLAMotor: Test-Guided Enhancement of Vision-Language-Action Models via Agent-BasedData Synthesis
  3. 2026Shifting Uncertainty to Critical Moments: Towards Reliable Uncertainty Quantification for VLA Model
  4. 2026FATE-VLA:Failue-aware test generation for vision-language-action models
  5. 2026Beyond In-Distribution Preservation: Recovering Generalization in Quantized VLAs via Vulnerability-Oriented Tuning
  6. 2026BOKBO (Best of K Bad Options): Calibrated Abstention for VLA Policies
  7. 2026Are Vision-Language-Action Models Robust to One-Step Observation Perturbations?
  8. 2026Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring