mlsanityData to 7 Oct 2026

Correcting Robot Plans with Natural Language Feedback

arXiv 2204.05186 ↗First posted 11 Apr 2022cs.RO cs.AI cs.CLcited by 68

Not on the map. Why?

Abstract (arXiv)

When humans design cost or goal specifications for robots, they often produce specifications that are ambiguous, underspecified, or beyond planners' ability to solve. In these cases, corrections provide a valuable tool for human-in-the-loop robot control. Corrections might take the form of new goal specifications, new constraints (e.g. to avoid specific objects), or hints for planning algorithms (e.g. to visit specific waypoints). Existing correction methods (e.g. using a joystick or direct manipulation of an end effector) require full teleoperation or real-time interaction. In this paper, we explore natural language as an expressive and flexible tool for robot correction. We describe how to map from natural language sentences to transformations of cost functions. We show that these transformations enable users to correct goals, update robot motions to accommodate additional user preferences, and recover from planning errors. These corrections can be leveraged to get 81% and 93% success rates on tasks where the original planner failed, with either one or two language corrections. Our method makes it possible to compose multiple constraints and generalizes to unseen scenes, objects, and sentences in simulated environments and real-world environments.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 18 of its 44 references that are in the map. First posted on arXiv 11 Apr 2022.

Builds on

18 of 44 references.

References · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  5. 2017Inverse Reward Designcited by 246
  6. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  7. 2021STORM: An Integrated Framework for Fast Joint-Space Model-Predictive Control for Reactive Manipulationcited by 46
  8. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  9. 2021Language Grounding with 3D Objectscited by 43
  10. 2018Guiding Policies with Language via Meta-Learningcited by 34
  11. 2016Learning Preferences for Manipulation Tasks from Online Coactive Feedbackcited by 31
  12. 2021LILA: Language-Informed Latent Actionscited by 27
6 more
  1. 2021NViSII: A Scriptable Tool for Photorealistic Image Generationcited by 24
  2. 2015Alignment-based compositional semantics for instruction followingcited by 21
  3. 2019Prospection: Interpretable Plans From Language By Predicting the Futurecited by 21
  4. 2017Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularitiescited by 19
  5. 2020Translating Natural Language Instructions to Computer Programs for Robot Manipulationcited by 5
  6. 2018Learning a Policy for Opportunistic Active Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 20; 2024: 22; 2025: 7; 2026: 11.

68 papers.

Sort:

By area:

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  5. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  6. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  7. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  8. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  9. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  10. 2024RT-H: Action Hierarchies Using Languagecited by 204
  11. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  12. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
38 more
  1. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  2. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  3. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  4. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  5. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  6. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  7. 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Modelscited by 33
  8. 2023Interactive Natural Language Processingcited by 31
  9. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  10. 2023Interactive Robot Learning from Verbal Correctioncited by 28
  11. 2023Interactive Planning Using Large Language Models for Partially Observable Robotics Taskscited by 27
  12. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  13. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  14. 2022How to talk so AI will learn: Instructions, descriptions, and autonomycited by 22
  15. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 20
  16. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  17. 2025VerifyLLM: LLM-Based Pre-Execution Task Plan Verification for Robotscited by 14
  18. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedbackcited by 13
  19. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  20. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9
  21. 2024Adaptive Language-Guided Abstraction from Contrastive Explanationscited by 8
  22. 2025IRef-VLA: A Benchmark for Interactive Referential Grounding with Imperfect Language in 3D Scenescited by 8
  23. 2023USA-Net: Unified Semantic and Affordance Representations for Robot Memorycited by 7
  24. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  25. 2023DiffVL: Scaling Up Soft Body Manipulation using Vision-Language Driven Differentiable Physicscited by 6
  26. 2026RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Roboticscited by 6
  27. 2023Architext: Language-Driven Generative Architecture Designcited by 5
  28. 2023Solving NLP Problems through Human-System Collaboration: A Discussion-based Approachcited by 5
  29. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  30. 2025KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulationcited by 5
  31. 2025ADAPT: Actively Discovering and Adapting to Preferences for any Taskcited by 5
  32. 2023Generating Language Corrections for Teaching Physical Control Taskscited by 4
  33. 2024Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaborationcited by 4
  34. 2024Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representationscited by 3
  35. 2025A Human-in-the-loop Approach to Robot Action Replanning through LLM Common-Sense Reasoningcited by 3
  36. 2026Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperationcited by 3
  37. 2022PARTNR: Pick and place Ambiguity Resolving by Trustworthy iNteractive leaRningcited by 2
  38. 2023Natural Language Robot Programming: NLP integrated with autonomous robotic graspingcited by 2

50 most cited of 68 All 68 (JSON)