mlsanityData to 14 Oct 2026

Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Corrections

arXiv 2311.10678 ↗First posted 17 Nov 2023cs.RO cs.AI cs.LGcited by 45

Not on the map. Why?

Abstract (arXiv)

Today's robot policies exhibit subpar performance when faced with the challenge of generalizing to novel environments. Human corrective feedback is a crucial form of guidance to enable such generalization. However, adapting to and learning from online human corrections is a non-trivial endeavor: not only do robots need to remember human feedback over time to retrieve the right information in new settings and reduce the intervention rate, but also they would need to be able to respond to feedback that can be arbitrary corrections about high-level human preferences to low-level adjustments to skill parameters. In this work, we present Distillation and Retrieval of Online Corrections (DROC), a large language model (LLM)-based system that can respond to arbitrary forms of language feedback, distill generalizable knowledge from corrections, and retrieve relevant past experiences based on textual and visual similarity for improving performance in novel settings. DROC is able to respond to a sequence of online language corrections that address failures in both high-level task plans and low-level skill primitives. We demonstrate that DROC effectively distills the relevant information from the sequence of online corrections in a knowledge base and retrieves that knowledge in settings with new task or object instances. DROC outperforms other techniques that directly generate robot code via LLMs by using only half of the total number of corrections needed in the first round and requires little to no corrections after two iterations. We show further results, videos, prompts and code on https://sites.google.com/stanford.edu/droc .

Key facts. Cited by 45 papers in the map (data through 14 Oct 2026). Builds on 40 of its 46 references that are in the map. First posted on arXiv 17 Nov 2023.

Builds on

40 of 46 references.

References · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2023Segment Anythingcited by 8,234
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  12. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
28 more
  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  3. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  4. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  6. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 611
  7. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  8. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  9. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  10. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  11. 2023Reward Design with Language Modelscited by 261
  12. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 224
  13. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  14. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  15. 2023Large Language Models as General Pattern Machinescited by 197
  16. 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correctioncited by 175
  17. 2022Language Models with Image Descriptors are Strong Few-Shot Video-Language Learnerscited by 117
  18. 2022Correcting Robot Plans with Natural Language Feedbackcited by 69
  19. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 63
  20. 2023DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignmentcited by 61
  21. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  22. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  23. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  24. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  25. 2018Guiding Policies with Language via Meta-Learningcited by 35
  26. 2021Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Taskscited by 32
  27. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformerscited by 17
  28. 2023Gesture-Informed Robot Assistance via Foundation Modelscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 19; 2026: 11.

45 papers.

Sort:

By area:

  1. 2024RT-H: Action Hierarchies Using Languagecited by 210
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  3. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  4. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 44
  5. 2024FlowRetrieval: Flow-Guided Data Retrieval for Few-Shot Imitation Learningcited by 33
  6. 2025DataMIL: Selecting Data for Robot Imitation Learning with Datamodelscited by 28
  7. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 25
  8. 2024A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulationcited by 21
  9. 2024AIC MLLM: Autonomous Interactive Correction MLLM for Robust Robotic Manipulationcited by 20
  10. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  11. 2024Automating Robot Failure Recovery Using Vision-Language Models With Optimized Promptscited by 16
  12. 2024Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Modelscited by 15
33 more
  1. 2024FLAIR: Feeding via Long-horizon AcquIsition of Realistic dishescited by 10
  2. 2025Distilling On-device Language Models for Robot Planning with Minimal Human Interventioncited by 10
  3. 2024A Survey on Human-AI Collaboration with Large Foundation Modelscited by 7
  4. 2025Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skillscited by 7
  5. 2025RT-Cache: Training-Free Retrieval for Real-Time Manipulationcited by 6
  6. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 5
  7. 2025KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulationcited by 5
  8. 2025COLLAGE: Adaptive Fusion-based Retrieval for Augmented Policy Learningcited by 5
  9. 2024Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaborationcited by 4
  10. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  11. 2025Human-like Navigation in a World Built for Humanscited by 4
  12. 2024Social Learning through Interactions with Other Agents: A Surveycited by 3
  13. 2025ViReSkill: Vision-Grounded Replanning with Skill Memory for LLM-Based Planning in Lifelong Robot Learningcited by 3
  14. 2025ELHPlan: Efficient Long-Horizon Task Planning for Multi-Agent Collaborationcited by 3
  15. 2025RAIDER: Tool-Equipped Large Language Model Agent for Robotic Action Issue Detection, Explanation and Recoverycited by 2
  16. 2025Learning Adaptive Dexterous Grasping from Single Demonstrationscited by 2
  17. 2025A Pragmatist Robot: Learning to Plan Tasks by Experiencing the Real Worldcited by 1
  18. 2025INTENTION: Inferring Tendencies of Humanoid Robot Motion Through Interactive Intuition and Grounded VLMcited by 1
  19. 2025LLM-Driven Self-Refinement for Embodied Drone Task Planningcited by 1
  20. 2025FT-NCFM: An Influence-Aware Data Distillation Framework for Efficient VLA Modelscited by 1
  21. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 1
  22. 2026Generalizing Manipulation Skills with a Local Coding Agentcited by 1
  23. 2026In-Context Learning for Robots: Methods and Applicationscited by 1
  24. 2024Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Modelscited by 0
  25. 2025LangPert: Detecting and Handling Task-level Perturbations for Robust Object Rearrangementcited by 0
  26. 2025iLearnRobot: An Interactive Learning-Based Multi-Modal Robot with Continuous Improvementcited by 0
  27. 2026Learning to Forget -- Hierarchical Episodic Memory for Lifelong Robot Deploymentcited by 0
  28. 2026SPINE: Bridging the Cyber-Physical Gap with Agentic AIcited by 0
  29. 2026What Language Does and What the Evidence Supports: A Functional Role Taxonomy and Evidence Audit of Language Grounding in Embodied Agentscited by 0
  30. 2026Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Modelscited by 0
  31. 2026When Should a Failing Robot Ask? Initiating Corrective Human-Robot Dialogue from Audited Sensor Evidencecited by 0
  32. 2026Skill-Space Shooting for Autonomous Robot Policy Improvementcited by 0
  33. 2026InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulationcited by 0