mlsanityData to 7 Oct 2026

Yell At Your Robot: Improving On-the-Fly from Language Corrections

arXiv 2403.12910 ↗First posted 19 Mar 2024cs.RO cs.AI cs.LGcited by 123

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Hierarchical policies that combine language and low-level control have been shown to perform impressively long-horizon robotic tasks, by leveraging either zero-shot high-level planners like pretrained language and vision-language models (LLMs/VLMs) or models trained on annotated robotic demonstrations. However, for complex and dexterous skills, attaining high success rates on long-horizon tasks still represents a major challenge -- the longer the task is, the more likely it is that some stage will fail. Can humans help the robot to continuously improve its long-horizon task performance through intuitive and natural feedback? In this paper, we make the following observation: high-level policies that index into sufficiently rich and expressive low-level language-conditioned skills can be readily supervised with human feedback in the form of language corrections. We show that even fine-grained corrections, such as small movements ("move a bit to the left"), can be effectively incorporated into high-level policies, and that such corrections can be readily obtained from humans observing the robot and making occasional suggestions. This framework enables robots not only to rapidly adapt to real-time language feedback, but also incorporate this feedback into an iterative training scheme that improves the high-level policy's ability to correct errors in both low-level execution and high-level decision-making purely from verbal feedback. Our evaluation on real hardware shows that this leads to significant performance improvement in long-horizon, dexterous manipulation tasks without the need for any additional teleoperation. Videos and code are available at https://yay-robot.github.io/.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 70 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

25 of 70 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  2. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  8. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  9. 2024RT-H: Action Hierarchies Using Languagecited by 204
  10. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  11. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  12. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 101
11 more
  1. 2021Skill Induction and Planning with Latent Languagecited by 99
  2. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  3. 2017Hierarchical and Interpretable Skill Acquisition in Multi-task Reinforcement Learningcited by 77
  4. 2017Learning with Latent Languagecited by 75
  5. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  6. 2023Multi-Stage Cable Routing through Hierarchical Imitation Learningcited by 59
  7. 2023Interactive Robot Learning from Verbal Correctioncited by 28
  8. 2024MOSAIC: Modular Foundation Models for Assistive and Interactive Cookingcited by 9
  9. 2023Learning Reusable Manipulation Strategiescited by 8
  10. 2018Multi-task Learning for Continuous Controlcited by 6
  11. 2023Hierarchical reinforcement learning with natural language subgoalscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 56; 2026: 31.

123 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  3. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  4. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  5. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  6. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
  7. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  8. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 105
  9. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  10. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  11. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 80
  12. 2024The Ingredients for Robotic Diffusion Transformerscited by 72
38 more
  1. 2025MemER: Scaling Up Memory for Robot Control via Experience Retrievalcited by 64
  2. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  3. 2025Actions as Language: Fine-Tuning VLMs into VLAs Without Catastrophic Forgettingcited by 52
  4. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 50
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  6. 2025AMO: Adaptive Motion Optimization for Hyper-Dexterous Humanoid Whole-Body Controlcited by 47
  7. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  8. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  9. 2025Embodied-Reasoner: Synergizing Visual Search, Reasoning, and Action for Embodied Interactive Taskscited by 43
  10. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  11. 2025Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulationcited by 40
  12. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  13. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  14. 2024Inference-Time Policy Steering through Human Interactionscited by 35
  15. 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Modelscited by 33
  16. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  17. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordancecited by 29
  18. 2024ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoningcited by 27
  19. 2025LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Taskscited by 25
  20. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  21. 2025SRT-H: A Hierarchical Framework for Autonomous Surgery via Language Conditioned Imitation Learningcited by 23
  22. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  23. 2024InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulationcited by 21
  24. 2025FPC-VLA: A Vision-Language-Action Framework with a Supervisor for Failure Prediction and Correctioncited by 21
  25. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 20
  26. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulationcited by 20
  27. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitationcited by 18
  28. 2024Automating Robot Failure Recovery Using Vision-Language Models With Optimized Promptscited by 16
  29. 2025Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalizationcited by 16
  30. 2025Learning Compositional Behaviors from Demonstration and Languagecited by 15
  31. 2025SwitchVLA: Execution-Aware Task Switching for Vision-Language-Action Modelscited by 15
  32. 2024doScenes: An Autonomous Driving Dataset with Natural Language Instruction for Human Interaction and Vision-Language Navigationcited by 14
  33. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
  34. 2025Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Modelscited by 13
  35. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
  36. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  37. 2025LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulationcited by 11
  38. 2026SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarioscited by 11

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026From Local Corrections to Generalized Skills: Improving Neuro-Symbolic Policies with MEMO
  2. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
  3. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  4. 2026Measuring Language Transfer in Robot Policies: Adding Greek to a Cosmos3 Vision-Language-Action Policy
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
  6. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  7. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulation
  8. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfer