mlsanityData to 7 Oct 2026

Language Conditioned Imitation Learning over Unstructured Data

arXiv 2005.07648 ↗First posted 15 May 2020cs.RO cs.AI cs.CLcited by 208

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

Natural language is perhaps the most flexible and intuitive way for humans to communicate tasks to a robot. Prior work in imitation learning typically requires each task be specified with a task id or goal image -- something that is often impractical in open-world environments. On the other hand, previous approaches in instruction following allow agent behavior to be guided by language, but typically assume structure in the observations, actuators, or language that limit their applicability to complex settings like robotics. In this work, we present a method for incorporating free-form natural language conditioning into imitation learning. Our approach learns perception from pixels, natural language understanding, and multitask continuous control end-to-end as a single neural network. Unlike prior work in imitation learning, our method is able to incorporate unlabeled and unstructured demonstration data (i.e. no task or language labels). We show this dramatically improves language conditioned performance, while reducing the cost of language annotation to less than 1% of total data. At test time, a single language conditioned visuomotor policy trained with our method can perform a wide variety of robotic manipulation skills in a 3D environment, specified only with natural language descriptions of each task (e.g. "open the drawer...now pick up the block...now press the green button..."). To scale up the number of instructions an agent can follow, we propose combining text conditioned policies with large pretrained neural language models. We find this allows a policy to be robust to many out-of-distribution synonym instructions, without requiring new demonstrations. See videos of a human typing live text commands to our agent at language-play.github.io

Key facts. Cited by 208 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 54 references that are in the map. First posted on arXiv 15 May 2020.

Builds on

36 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation25 Nov

Other references · 35

  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2017Hindsight Experience Replaycited by 1,096
  3. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  6. 2018A Survey on Deep Transfer Learningcited by 417
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  9. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  10. 2019Learning Latent Plans from Playcited by 310
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  12. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
23 more
  1. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  2. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  3. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  4. 2019Multilingual Universal Sentence Encoder for Semantic Retrievalcited by 192
  5. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  6. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  7. 2019Goal-conditioned Imitation Learningcited by 159
  8. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  9. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  10. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  11. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  12. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  13. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  14. 2017Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learningcited by 97
  15. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  16. 2019Wasserstein Dependency Measure for Representation Learningcited by 69
  17. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  18. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  19. 2019RTFM: Generalising to Novel Environment Dynamics via Readingcited by 33
  20. 2018Interactive Grounded Language Acquisition and Generalization in a 2D Worldcited by 21
  21. 2020Scalable Multi-Task Imitation Learning with Autonomous Improvementcited by 21
  22. 2019Self-supervised Learning of Image Embedding for Continuous Controlcited by 16
  23. 2019Online Object Representations with Contrastive Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 39; 2023: 55; 2024: 58; 2025: 35; 2026: 16.

208 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  3. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  5. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  6. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  8. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  9. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  10. 2023Learning Interactive Real-World Simulatorscited by 440
  11. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  12. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
38 more
  1. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  2. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  3. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  4. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  5. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  6. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  7. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  8. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  9. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  10. 2022Multi-Game Decision Transformerscited by 212
  11. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  12. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  13. 2023RH20T: A Comprehensive Robotic Dataset for Learning Diverse Skills in One-Shotcited by 173
  14. 2023RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunkingcited by 167
  15. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  16. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  17. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  18. 2023Large Language Models for Robotics: A Surveycited by 132
  19. 2023RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketchescited by 131
  20. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  21. 2025Unified Vision-Language-Action Modelcited by 128
  22. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  23. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  24. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  25. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  26. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  27. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agentscited by 101
  28. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  29. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  30. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  31. 2022"This is my unicorn, Fluffy": Personalizing frozen vision-language representationscited by 95
  32. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  33. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  34. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  35. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  36. 2024GaussianGrasper: 3D Language Gaussian Splatting for Open-vocabulary Robotic Graspingcited by 77
  37. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  38. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 76

50 most cited of 208 All 208 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks
  2. 2020Translating Natural Language Instructions to Computer Programs for Robot Manipulation
  3. 2024Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning
  4. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  5. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  6. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  7. 2024Learning with Language-Guided State Abstractions
  8. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity