mlsanityData to 7 Oct 2026

Language-Conditioned Imitation Learning for Robot Manipulation Tasks

arXiv 2010.12083 ↗First posted 22 Oct 2020cs.RO cs.CL cs.CVcited by 136

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Imitation learning is a popular approach for teaching motor skills to robots. However, most approaches focus on extracting policy parameters from execution traces alone (i.e., motion trajectories and perceptual data). No adequate communication channel exists between the human expert and the robot to describe critical aspects of the task, such as the properties of the target object or the intended shape of the motion. Motivated by insights into the human teaching process, we introduce a method for incorporating unstructured natural language into imitation learning. At training time, the expert can provide demonstrations along with verbal descriptions in order to describe the underlying intent (e.g., "go to the large green bowl"). The training process then interrelates these two modalities to encode the correlations between language, perception, and motion. The resulting language-conditioned visuomotor policies can be conditioned at runtime on new human commands and instructions, which allows for more fine-grained control over the trained policies while also reducing situational ambiguity. We demonstrate in a set of simulation experiments how our approach can learn language-conditioned manipulation policies for a seven-degree-of-freedom robot arm and compare the results to a variety of alternative methods.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 17 of its 34 references that are in the map. First posted on arXiv 22 Oct 2020.

Builds on

17 of 34 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  6. 2016Language Modeling with Gated Convolutional Networkscited by 688
  7. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  8. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  9. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  10. 2019Goal-conditioned Imitation Learningcited by 159
  11. 2019PyRep: Bringing V-REP to Deep Robot Learningcited by 43
  12. 2019Chasing Ghosts: Instruction Following as Bayesian State Trackingcited by 41
5 more
  1. 2020The EMPATHIC Framework for Task Learning from Implicit Human Feedbackcited by 38
  2. 2018Pay attention! - Robustifying a Deep Visuomotor Policy through Task-Focused Attentioncited by 11
  3. 2020Deep compositional robotic planners that follow natural language commandscited by 10
  4. 2019Probabilistic Multimodal Modeling for Human-Robot Interaction Taskscited by 6
  5. 2019Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Controlcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 19; 2023: 39; 2024: 34; 2025: 24; 2026: 13.

136 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  5. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  6. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  7. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  8. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  9. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  10. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  11. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  12. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
38 more
  1. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  2. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
  3. 2024RT-H: Action Hierarchies Using Languagecited by 204
  4. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  5. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  6. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  7. 2023Language-Driven Representation Learning for Roboticscited by 163
  8. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  9. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 149
  10. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  11. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  12. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  13. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  14. 2024HiRT: Enhancing Robotic Control with Hierarchical Robot Transformerscited by 72
  15. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 69
  16. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  17. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  18. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  19. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  20. 2023PolarNet: 3D Point Clouds for Language-Guided Robotic Manipulationcited by 56
  21. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48
  22. 2022LATTE: LAnguage Trajectory TransformErcited by 40
  23. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  24. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  25. 2024Towards Generalizable Vision-Language Robotic Manipulation: A Benchmark and LLM-guided 3D Policycited by 37
  26. 2024From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Controlcited by 34
  27. 2023A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Cluttercited by 33
  28. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agentscited by 32
  29. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  30. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  31. 2022LISA: Learning Interpretable Skill Abstractions from Languagecited by 30
  32. 2026RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalizationcited by 29
  33. 2021LILA: Language-Informed Latent Actionscited by 27
  34. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulationcited by 26
  35. 2024SUGAR: Pre-training 3D Visual Representations for Roboticscited by 26
  36. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  37. 2023InsActor: Instruction-driven Physics-based Characterscited by 23
  38. 2023Language-guided Robot Grasping: CLIP-based Referring Grasp Synthesis in Cluttercited by 22

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Language Conditioned Imitation Learning over Unstructured Data
  2. 2020Translating Natural Language Instructions to Computer Programs for Robot Manipulation
  3. 2024Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning
  4. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  5. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  6. 2019Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration
  7. 2024Learning with Language-Guided State Abstractions
  8. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics