mlsanityData to 7 Oct 2026

SAFE: Multitask Failure Detection for Vision-Language-Action Models

arXiv 2506.09937 ↗First posted 11 Jun 2025cs.RO cs.AIcited by 74

On the map: Vision-language-action models → Failure recovery and robustness

Abstract (arXiv)

While vision-language-action models (VLAs) have shown promising robotic behaviors across a diverse set of manipulation tasks, they achieve limited success rates when deployed on novel tasks out of the box. To allow these policies to safely interact with their environments, we need a failure detector that gives a timely alert such that the robot can stop, backtrack, or ask for help. However, existing failure detectors are trained and tested only on one or a few specific tasks, while generalist VLAs require the detector to generalize and detect failures also in unseen tasks and novel environments. In this paper, we introduce the multitask failure detection problem and propose SAFE, a failure detector for generalist robot policies such as VLAs. We analyze the VLA feature space and find that VLAs have sufficient high-level knowledge about task success and failure, which is generic across different tasks. Based on this insight, we design SAFE to learn from VLA internal features and predict a single scalar indicating the likelihood of task failure. SAFE is trained on both successful and failed rollouts and is evaluated on unseen tasks. SAFE is compatible with different policy architectures. We test it on OpenVLA, π0, and π0-FAST in both simulated and real-world environments extensively. We compare SAFE with diverse baselines and show that SAFE achieves state-of-the-art failure detection performance and the best trade-off between accuracy and detection time using conformal prediction. More qualitative results and code can be found at the project webpage: https://vla-safe.github.io/

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Failure recovery and robustness, one of 13 directions in Vision-language-action models. Builds on 61 of its 77 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

61 of 77 references.

References · 61

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  6. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  10. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  11. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,338
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
38 more
  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  4. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  6. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  8. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  9. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  10. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 643
  11. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  12. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  13. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 360
  14. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  15. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  16. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  17. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  18. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  19. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  20. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  21. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  22. 2023Vision-Language Models as Success Detectorscited by 138
  23. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  24. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 118
  25. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  26. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  27. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 74
  28. 2022Sequential Predictive Conformal Inference for Time Seriescited by 71
  29. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 67
  30. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  31. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulationcited by 59
  32. 2020Conformal prediction for time seriescited by 53
  33. 2025Steer LLM Latents for Hallucination Detectioncited by 49
  34. 2025Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policiescited by 49
  35. 2022Normalizing flow neural networks by JKO schemecited by 39
  36. 2020Out-of-Distribution Detection for Automotive Perceptioncited by 37
  37. 2023Model-Based Runtime Monitoring with Interactive Imitation Learningcited by 37
  38. 2022A System-Level View on Out-of-Distribution Data in Roboticscited by 33

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 62.

74 papers.

Sort:

By area:

  1. 2025LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorizationcited by 166
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  4. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  5. 2025Verifier-free Test-Time Sampling for Vision-Language-Action Modelscited by 21
  6. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  7. 2025FreezeVLA: Action-Freezing Attacks against Vision-Language-Action Modelscited by 19
  8. 2026Scaling Verification Can Be More Effective than Scaling Policy Learning for Vision-Language-Action Alignmentcited by 15
  9. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
  10. 2026CLARE: Continual Learning for Vision-Language-Action Models via Autonomous Adapter Routing and Expansioncited by 14
  11. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  12. 2026Vision-Language-Action Safety: Threats, Challenges, Evaluations, and Mechanismscited by 12
38 more
  1. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 11
  2. 2025SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Modelcited by 10
  3. 2025Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulationcited by 8
  4. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  5. 2026Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latentscited by 8
  6. 2025ExpReS-VLA: Specializing Vision-Language-Action Models Through Experience Replay and Retrievalcited by 7
  7. 2026Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesiscited by 7
  8. 2026See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulationcited by 6
  9. 2026Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoringcited by 6
  10. 2026ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Controlcited by 5
  11. 2026Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steeringcited by 4
  12. 2026Uncertainty Quantification for Flow-Based Vision-Language-Action Modelscited by 4
  13. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 4
  14. 2026SCALE: Self-uncertainty Conditioned Adaptive Looking and Execution for Vision-Language-Action Modelscited by 3
  15. 2026Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Modelscited by 3
  16. 2026Tri-Info: Generalizable, Interpretable Failure Prediction for VLA Models via Information Theorycited by 3
  17. 2026Failure Detection for Surgical Robot Imitation Policies via Flow-Matching World Modelingcited by 3
  18. 2026Decoding Task Progress from VLA Representationscited by 3
  19. 2025EVE: A Generator-Verifier System for Generative Policiescited by 2
  20. 2026AEGIS: A Backup Reflex for Physical AIcited by 2
  21. 2026ActProbe: Action-Space Probe for Early Failure Detection of Generative Robot Policiescited by 2
  22. 2026PATCH: Action-Chunk-Conditioned Latent Patch Innovation Monitoring for Robot Manipulationcited by 2
  23. 2026SafeDojo: Safe Reinforcement Learning for VLA via Interactive World Modelcited by 2
  24. 2026VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Modelscited by 2
  25. 2026Early Warning Signals for OpenVLA Failure under Visual Distribution Shiftcited by 2
  26. 2026SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibrationcited by 2
  27. 2026ContactGuard: Pre-Contact Execution Monitoring with Action-Conditioned Latent World Modelscited by 2
  28. 2026EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agentscited by 2
  29. 2026V-VLAPS: Value-Guided Planning for Vision-Language-Action Modelscited by 1
  30. 2026Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Modelscited by 1
  31. 2026Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Modelscited by 1
  32. 2026DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policiescited by 1
  33. 2026VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Modelscited by 1
  34. 2026FATE-VLA:Failue-aware test generation for vision-language-action modelscited by 1
  35. 2026RoboBRIDGE: A Modular Framework for Bridging Policies to Robust Real-World Robotic Agentscited by 1
  36. 2026Uncovering and Mitigating Positional Blind Spots in Vision-Language-Action Modelscited by 1
  37. 2026ValueFormer: A Causal Transformer Value Function with Stage-Aware Labels for Semi-Autonomous Vision-Language-Action Policiescited by 1
  38. 2026Act, Think or Abstain: Complexity-Aware Adaptive Inference for Vision-Language-Action Modelscited by 0

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026FailureSpot: Label-Efficient Timestamp-Level Failure Detection for Vision-Language-Action Models
  2. 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
  3. 2026SafeVLA-Bench: A Benchmark for the Success-Safety Gap in Vision-Language-Action Models
  4. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models
  5. 2026How VLAs (Really) Work In Open-World Environments
  6. 2026ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models
  7. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learning
  8. 2025FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correction