mlsanityData to 7 Oct 2026

"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomy

arXiv 2301.02555 ↗First posted 6 Jan 2023cs.RO cs.AI cs.CLcited by 60

Not on the map. Why?

Abstract (arXiv)

Systems for language-guided human-robot interaction must satisfy two key desiderata for broad adoption: adaptivity and learning efficiency. Unfortunately, existing instruction-following agents cannot adapt, lacking the ability to incorporate online natural language supervision, and even if they could, require hundreds of demonstrations to learn even simple policies. In this work, we address these problems by presenting Language-Informed Latent Actions with Corrections (LILAC), a framework for incorporating and adapting to natural language corrections - "to the right," or "no, towards the book" - online, during execution. We explore rich manipulation domains within a shared autonomy paradigm. Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot: language is an input to a learned model that produces a meaningful, low-dimensional control space that the human can use to guide the robot. Each real-time correction refines the human's control space, enabling precise, extended behaviors - with the added benefit of requiring only a handful of demonstrations to learn. We evaluate our approach via a user study where users work with a Franka Emika Panda manipulator to complete complex manipulation tasks. Compared to existing learned baselines covering both open-loop instruction following and single-turn shared autonomy, we show that our corrections-aware approach obtains higher task completion rates, and is subjectively preferred by users because of its reliability, precision, and ease of use.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 6 Jan 2023.

Builds on

37 of 54 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
25 more
  1. 2019Vision-and-Dialog Navigationcited by 246
  2. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  3. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  4. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  5. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  6. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  7. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  8. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 98
  9. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  10. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  11. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  12. 2017Shared Autonomy via Hindsight Optimization for Teleoperation and Teamingcited by 39
  13. 2018Guiding Policies with Language via Meta-Learningcited by 34
  14. 2020Shared Autonomy with Learned Latent Actionscited by 34
  15. 2019Controlling Assistive Robots with Learned Latent Actionscited by 28
  16. 2021LILA: Language-Informed Latent Actionscited by 27
  17. 2019Improving Grounded Natural Language Understanding through Human-Robot Dialogcited by 25
  18. 2020Learning Adaptive Language Interfaces through Decompositioncited by 22
  19. 2021Learning Human Objectives from Sequences of Physical Correctionscited by 21
  20. 2017Accurately and Efficiently Interpreting Human-Robot Instructions of Varying Granularitiescited by 19
  21. 2021Learning Latent Actions to Control Assistive Robotscited by 17
  22. 2022Reshaping Robot Trajectories Using Natural Language Commands: A Study of Multi-Modal Data Alignment Using Transformerscited by 17
  23. 2020Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Datacited by 16
  24. 2021Learning Visually Guided Latent Actions for Assistive Teleoperationcited by 12
  25. 2021Learning Online from Corrective Feedback: A Meta-Algorithm for Roboticscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 20; 2025: 17; 2026: 13.

60 papers.

Sort:

By area:

  1. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  2. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  3. 2024RT-H: Action Hierarchies Using Languagecited by 204
  4. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 171
  5. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  6. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  7. 2024Unpacking Failure Modes of Generative Policies: Runtime Monitoring of Consistency and Progresscited by 63
  8. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  9. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 49
  10. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  11. 2023Distilling and Retrieving Generalizable Knowledge for Robot Manipulation via Language Correctionscited by 44
  12. 2023Chat with the Environment: Interactive Multimodal Perception Using Large Language Modelscited by 42
38 more
  1. 2024RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketchescited by 41
  2. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  3. 2023Think, Act, and Ask: Open-World Interactive Personalized Robot Navigationcited by 31
  4. 2023LLF-Bench: Benchmark for Interactive Learning from Language Feedbackcited by 30
  5. 2023Interactive Planning Using Large Language Models for Partially Observable Robotics Taskscited by 27
  6. 2024DriVLMe: Enhancing LLM-based Autonomous Driving Agents with Embodied and Social Experiencescited by 27
  7. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  8. 2024Language-Conditioned Robotic Manipulation with Fast and Slow Thinkingcited by 17
  9. 2024Object-Centric Instruction Augmentation for Robotic Manipulationcited by 16
  10. 2024P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Taskcited by 14
  11. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  12. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
  13. 2025Human-assisted Robotic Policy Refinement via Action Preference Optimizationcited by 13
  14. 2025Guiding Data Collection via Factored Scaling Curvescited by 10
  15. 2025Growing with Your Embodied Agent: A Human-in-the-Loop Lifelong Code Generation Framework for Long-Horizon Manipulation Skillscited by 6
  16. 2024Leveraging Large Language Models in Human-Robot Interaction: A Critical Analysis of Potential and Pitfallscited by 5
  17. 2025HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Modelscited by 5
  18. 2023Generating Language Corrections for Teaching Physical Control Taskscited by 4
  19. 2024Vocal Sandbox: Continual Learning and Adaptation for Situated Human-Robot Collaborationcited by 4
  20. 2025Shared Autonomy for Proximal Teachingcited by 4
  21. 2025Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agentcited by 4
  22. 2025Human-like Navigation in a World Built for Humanscited by 4
  23. 2025Masked IRL: LLM-Guided Reward Disambiguation from Demonstrations and Languagecited by 4
  24. 2025DuetUI: A Bidirectional Context Loop for Human-Agent Co-Generation of Task-Oriented Interfacescited by 3
  25. 2026Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperationcited by 3
  26. 2024LLM Granularity for On-the-Fly Robot Controlcited by 2
  27. 2024Robot Behavior Personalization from Sparse User Feedbackcited by 2
  28. 2025ProVox: Personalization and Proactive Planning for Situated Human-Robot Collaborationcited by 2
  29. 2026Causally Robust Reward Learning from Reason-Augmented Preference Feedbackcited by 2
  30. 2024Enabling robots to follow abstract instructions and complete complex dynamic taskscited by 1
  31. 2024Scaffolded Language Models with Language Supervision for Mixed-Autonomy: A Surveycited by 1
  32. 2025Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Modelscited by 1
  33. 2026CANINE: Coaching Visually Impaired Users for Interactive Navigation with a Robot Guide Dogcited by 1
  34. 2026A Few Words Go a Long Way: Language Guided Robot Policy Synthesiscited by 1
  35. 2026Robots Influencing Humans to Reveal their Goals during Collaboration and Competitioncited by 1
  36. 2024Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Modelscited by 0
  37. 2024Skill Generalization with Verbscited by 0
  38. 2024Adaptive Querying for Reward Learning from Human Feedbackcited by 0

50 most cited of 60 All 60 (JSON)