mlsanityData to 14 Oct 2026

AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation

arXiv 2410.00371 ↗First posted 1 Oct 2024cs.ROcited by 125

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Robotic manipulation in open-world settings requires not only task execution but also the ability to detect and learn from failures. While recent advances in vision-language models (VLMs) and large language models (LLMs) have improved robots' spatial reasoning and problem-solving abilities, they still struggle with failure recognition, limiting their real-world applicability. We introduce AHA, an open-source VLM designed to detect and reason about failures in robotic manipulation using natural language. By framing failure detection as a free-form reasoning task, AHA identifies failures and provides detailed, adaptable explanations across different robots, tasks, and environments. We fine-tuned AHA using FailGen, a scalable framework that generates the first large-scale dataset of robotic failure trajectories, the AHA dataset. FailGen achieves this by procedurally perturbing successful demonstrations from simulation. Despite being trained solely on the AHA dataset, AHA generalizes effectively to real-world failure datasets, robotic systems, and unseen tasks. It surpasses the second-best model (GPT-4o in-context learning) by 10.3% and exceeds the average performance of six compared models including five state-of-the-art VLMs by 35.3% across multiple metrics and datasets. We integrate AHA into three manipulation frameworks that utilize LLMs/VLMs for reinforcement learning, task and motion planning, and zero-shot trajectory generation. AHA's failure feedback enhances these policies' performances by refining dense reward functions, optimizing task planning, and improving sub-task verification, boosting task success rates by an average of 21.4% across all three tasks compared to GPT-4 models.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 10 of its 61 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

10 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation3 Sep
  2. 2024Octopi: Object Property Reasoning with Large Tactile-Language Models5 May
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023NEWTON: Are Large Language Models Capable of Physical Reasoning?10 Oct

Other references · 6

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  3. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 125
  6. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 45; 2026: 75.

125 papers.

Sort:

By area:

  1. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 197
  2. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 156
  3. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  4. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  5. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 80
  6. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 74
  7. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  8. 2025From Foresight to Forethought: VLM-In-the-Loop Policy Steering via Latent Alignmentcited by 52
  9. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  10. 2025Failure Prediction at Runtime for Generative Robot Policiescited by 42
  11. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 41
  12. 2026World-VLA-Loop: Closed-Loop Learning of Video World Model and VLA Policycited by 37
38 more
  1. 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Modelscited by 35
  2. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 33
  3. 2025CorrectNav: Self-Correction Flywheel Empowers Vision-Language-Action Navigation Modelcited by 27
  4. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 25
  5. 2025Safety Aware Task Planning via Large Language Models in Roboticscited by 24
  6. 2025FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correctioncited by 21
  7. 2026Large Language Model Reasoning Failurescited by 21
  8. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 20
  9. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  10. 2026PROGRESSLM: Towards Progress Reasoning in Vision-Language Modelscited by 15
  11. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Modelscited by 15
  12. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 14
  13. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
  14. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 14
  15. 2025Modeling Multiple Normal Action Representations for Error Detection in Procedural Taskscited by 12
  16. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Modelscited by 12
  17. 2026Foresight: Failure Detection for Long-Horizon Robotic Manipulation with Action-Conditioned World Model Latentscited by 11
  18. 2025SeqVLA: Sequential Task Execution for Long-Horizon Manipulation with Completion-Aware Vision-Language-Action Modelcited by 10
  19. 2025Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Groundingcited by 10
  20. 2026Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesiscited by 10
  21. 2025Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robotscited by 9
  22. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 9
  23. 2026Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoringcited by 9
  24. 2026Towards the Harness of Embodied Agentscited by 9
  25. 2025Score the Steps, Not Just the Goal: VLM-Based Subgoal Evaluation for Robotic Manipulationcited by 8
  26. 2024RoboMD: Uncovering Robot Vulnerabilities through Semantic Potential Fieldscited by 7
  27. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  28. 2026Self-Refining Vision Language Model for Robotic Failure Detection and Reasoningcited by 7
  29. 2026See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulationcited by 7
  30. 2026From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulationcited by 7
  31. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 6
  32. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 5
  33. 2025ViPlan: A Benchmark for Visual Planning with Symbolic Predicates and Vision-Language Modelscited by 5
  34. 2026ReViP: Mitigating False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalancecited by 5
  35. 2026ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Controlcited by 5
  36. 2026FAR: Failure-Aware Retry for Test-Time Recovery and Continual Policy Improvementcited by 5
  37. 2025A Unified Framework for Real-Time Failure Handling in Robotics Using Vision-Language Models, Reactive Planner and Behavior Treescited by 4
  38. 2025Humanoid World Models: Open World Foundation Models for Humanoid Roboticscited by 4

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction
  2. 2026RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation ↗
  3. 2024Addressing Failures in Robotics using Vision-Based Language Models (VLMs) and Behavior Trees (BT)
  4. 2024Automating Robot Failure Recovery Using Vision-Language Models With Optimized Prompts
  5. 2026RoboProcessBench: Benchmarking Process-Aware Understanding in Vision-Language Robotic Manipulation
  6. 2026FLARE: A Failure-Aware Framework for Autonomous Correction and Recovery in Visual-Language Robotic Manipulation
  7. 2026VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models
  8. 2026Cracking Mobile Manipulation: From Failure Diagnosis to Agentic Robot System