mlsanityData to 14 Oct 2026

Interactive Robot Learning from Verbal Correction

arXiv 2310.17555 ↗First posted 26 Oct 2023cs.RO cs.AI cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

The ability to learn and refine behavior after deployment has become ever more important for robots as we design them to operate in unstructured environments like households. In this work, we design a new learning system based on large language model (LLM), OLAF, that allows everyday users to teach a robot using verbal corrections when the robot makes mistakes, e.g., by saying "Stop what you're doing. You should move closer to the cup." A key feature of OLAF is its ability to update the robot's visuomotor neural policy based on the verbal feedback to avoid repeating mistakes in the future. This is in contrast to existing LLM-based robotic systems, which only follow verbal commands or corrections but not learn from them. We demonstrate the efficacy of our design in experiments where a user teaches a robot to perform long-horizon manipulation tasks both in simulation and on physical hardware, achieving on average 20.0% improvement in policy success rate. Videos and more results are at https://ut-austin-rpl.github.io/olaf/

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

27 of 35 references.

References · 27

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  3. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  5. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  7. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  8. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  9. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  10. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  11. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 299
  12. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 289
15 more
  1. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 266
  2. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 241
  3. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  4. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  5. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 112
  6. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 108
  7. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  8. 2022Correcting Robot Plans with Natural Language Feedbackcited by 69
  9. 2021Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulationcited by 59
  10. 2021LazyDAgger: Reducing Context Switching in Interactive Imitation Learningcited by 51
  11. 2023Learning to Generate Better Than Your LLMcited by 49
  12. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  13. 2022PATO: Policy Assisted TeleOperation for Scalable Robot Data Collectioncited by 19
  14. 2016Comparing Human-Centric and Robot-Centric Sampling for Robot Deep Learning from Demonstrationscited by 17
  15. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 10; 2026: 9.

29 papers.

Sort:

By area:

  1. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  3. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 124
  4. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thoughtcited by 55
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  6. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 33
  7. 2024Autoregressive Action Sequence Learning for Robotic Manipulationcited by 27
  8. 2024Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptionscited by 23
  9. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  10. 2025Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbolscited by 14
  11. 2025Formalizing Learning from Language Feedback with Provable Guaranteescited by 13
  12. 2026See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulationcited by 7
17 more
  1. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  2. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  3. 2025Ask-to-Clarify: Resolving Instruction Ambiguity through Multi-turn Dialoguecited by 4
  4. 2025Human-like Navigation in a World Built for Humanscited by 4
  5. 2024From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledgecited by 3
  6. 2024Enhancing Q-Learning with Large Language Model Heuristicscited by 2
  7. 2025LLM-based Interactive Imitation Learning for Robotic Manipulationcited by 2
  8. 2025Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policiescited by 2
  9. 2026Benchmarking Affordance Generalization with BusyBoxcited by 2
  10. 2026IROSA: Interactive Robot Skill Adaptation using Natural Languagecited by 1
  11. 2026ORPA: Online Residual Policy Adaptation for Robot Manipulation Control with Human Feedbackcited by 1
  12. 2026Ludiscriptscriptstyle 0.1: An Agentic System for Socially Intelligent Robotscited by 1
  13. 2024The Power of Combined Modalities in Interactive Robot Learningcited by 0
  14. 2025CueLearner: Bootstrapping and local policy adaptation from relative feedbackcited by 0
  15. 2026Off-Policy Actor-Critic with Sigmoid-Bounded Entropy for Real-World Robot Learningcited by 0
  16. 2026Robust Instruction Compliance in Cooperative Multi-Agent Reinforcement Learningcited by 0
  17. 2026Future Anchored Verification and Online Recovery for World Action Modelscited by 0