mlsanityData to 7 Oct 2026

RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learning

arXiv 2409.14674 ↗First posted 23 Sep 2024cs.RO cs.CL cs.CVcited by 46

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Developing robust and correctable visuomotor policies for robotic manipulation is challenging due to the lack of self-recovery mechanisms from failures and the limitations of simple language instructions in guiding robot actions. To address these issues, we propose a scalable data generation pipeline that automatically augments expert demonstrations with failure recovery trajectories and fine-grained language annotations for training. We then introduce Rich languAge-guided failure reCovERy (RACER), a supervisor-actor framework, which combines failure recovery data with rich language descriptions to enhance robot control. RACER features a vision-language model (VLM) that acts as an online supervisor, providing detailed language guidance for error correction and task execution, and a language-conditioned visuomotor policy as an actor to predict the next actions. Our experimental results show that RACER outperforms the state-of-the-art Robotic View Transformer (RVT) on RLbench across various evaluation settings, including standard long-horizon tasks, dynamic goal-change tasks and zero-shot unseen tasks, achieving superior performance in both simulated and real world environments. Videos and code are available at: https://rich-language-failure-recovery.github.io.

Key facts. Cited by 46 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 46 references that are in the map. First posted on arXiv 23 Sep 2024.

Builds on

39 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Model-Based Runtime Monitoring with Interactive Imitation Learning26 Oct

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
26 more
  1. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  3. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  4. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  5. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  6. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  7. 2024RT-H: Action Hierarchies Using Languagecited by 204
  8. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 171
  9. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  10. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 135
  11. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 124
  12. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  13. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
  14. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  15. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
  16. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  17. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  18. 2021Transferable Dialogue Systems and User Simulatorscited by 35
  19. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  20. 2021LILA: Language-Informed Latent Actionscited by 27
  21. 2023CCIL: Continuity-based Data Augmentation for Corrective Imitation Learningcited by 26
  22. 2024IntervenGen: Interventional Data Generation for Robust and Data-Efficient Robot Imitation Learningcited by 24
  23. 2024Natural Language Can Help Bridge the Sim2Real Gapcited by 21
  24. 2024Recover: A Neuro-Symbolic Framework for Failure Detection and Recoverycited by 15
  25. 2024Evaluating Uncertainty-based Failure Detection for Closed-Loop LLM Plannerscited by 15
  26. 2024NaturalVLM: Leveraging Fine-grained Natural Language for Affordance-Guided Visual Manipulationcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 19; 2026: 24.

46 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  3. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  4. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  5. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  6. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 25
  7. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  8. 2025CUPID: Curating Data your Robot Loves with Influence Functionscited by 21
  9. 2025FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correctioncited by 21
  10. 2025Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correctioncited by 16
  11. 2025SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Modelscited by 15
  12. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
34 more
  1. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Modelscited by 14
  2. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  3. 2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Usecited by 7
  4. 2024Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learningcited by 6
  5. 2026See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulationcited by 6
  6. 2026From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulationcited by 6
  7. 2026CoMaTrack: Competitive Multi-Agent Game-Theoretic Tracking with Vision-Language-Action Modelscited by 5
  8. 2025A Unified Framework for Real-Time Failure Handling in Robotics Using Vision-Language Models, Reactive Planner and Behavior Treescited by 4
  9. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  10. 2026Sentinel-VLA: A Metacognitive VLA Model with Active Status Monitoring for Dynamic Reasoning and Error Recoverycited by 4
  11. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  12. 2026HeRO: Hierarchical 3D Semantic Representation for Pose-aware Object Manipulationcited by 3
  13. 2026Back to the Familiar Future: Failure Recovery for VLA Policies via Pre-Imagined Milestone Selectioncited by 3
  14. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulationcited by 2
  15. 2025RDD: Retrieval-Based Demonstration Decomposer for Planner Alignment in Long-Horizon Taskscited by 2
  16. 2025EVE: A Generator-Verifier System for Generative Policiescited by 2
  17. 2026InterPReT: Interactive Policy Restructuring and Training Enable Effective Imitation Learning from Laypersonscited by 2
  18. 2026Causally Robust Reward Learning from Reason-Augmented Preference Feedbackcited by 2
  19. 2026ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policiescited by 2
  20. 2025VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing, Speaking, and Actingcited by 1
  21. 2026Imagining Recovery: Inference-Time Counterfactual Realignment for Vision-Language-Action Modelscited by 1
  22. 2025TubeDAgger: Reducing the Number of Expert Interventions with Stochastic Reach-Tubescited by 0
  23. 2025BINDER: Instantly Adaptive Mobile Manipulation with Open-Vocabulary Commandscited by 0
  24. 2026Instrumentation for Imitation Learning: Enhancing Training Datasets for Clothes Hanger Insertioncited by 0
  25. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
  26. 2026SPINE: Bridging the Cyber-Physical Gap with Agentic AIcited by 0
  27. 2026Ontology-Grounded World Models for Failure Diagnosis and Closed-Loop Repair in Physical AI Systemscited by 0
  28. 2026Beyond Patch Removal: Persistent Adversarial Effects in Vision-Language-Action Policiescited by 0
  29. 2026When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidencecited by 0
  30. 2026FRAMES: Failure Recovery And Monitoring of Embodied Skills for Humanoid Loco-Manipulationcited by 0
  31. 2026CARE: Experience-Guided Atomic Corrective Execution for Vision-Language-Action Policiescited by 0
  32. 2026Kintsugi-VLA: Turning Failed Robot Rollouts into Recovery Data through Interventional Recoverabilitycited by 0
  33. 2026Spotter: Let the Embodied Model Lead, and the VLM Reflect for Itcited by 0
  34. 2026Multi-Granularity Language-Guided Imitation Learning via Instruction Decompositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis
  2. 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
  3. 2026Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models
  4. 2026FLARE: A Failure-Aware Framework for Autonomous Correction and Recovery in Visual-Language Robotic Manipulation
  5. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  6. 2025I-FailSense: Towards General Robotic Failure Detection with Vision-Language Models
  7. 2026Rewind-IL: Online Failure Detection and State Respawning for Imitation Learning
  8. 2024Adapt2Reward: Adapting Video-Language Models to Generalizable Robotic Rewards via Failure Prompts