mlsanityData to 7 Oct 2026

What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data

arXiv 2204.06252 ↗First posted 13 Apr 2022cs.RO cs.AI cs.CLcited by 137

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

A long-standing goal in robotics is to build robots that can perform a wide range of daily tasks from perceptions obtained with their onboard sensors and specified only via natural language. While recently substantial advances have been achieved in language-driven robotics by leveraging end-to-end learning from pixels, there is no clear and well-understood process for making various design choices due to the underlying variation in setups. In this paper, we conduct an extensive study of the most critical challenges in learning language conditioned policies from offline free-form imitation datasets. We further identify architectural and algorithmic techniques that improve performance, such as a hierarchical decomposition of the robot control learning, a multimodal transformer encoder, discrete latent plans and a self-supervised contrastive loss that aligns video and language representations. By combining the results of our investigation with our improved model components, we are able to present a novel approach that significantly outperforms the state of the art on the challenging language conditioned long-horizon robot manipulation CALVIN benchmark. We have open-sourced our implementation to facilitate future research in learning to perform many complex manipulation skills in a row specified with natural language. Codebase and trained models available at http://hulc.cs.uni-freiburg.de

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 53 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

42 of 53 references.

Closest ancestors · 2

Same direction, newest first

  1. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks22 Oct
  2. 2020Language Conditioned Imitation Learning over Unstructured Data15 May

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2020Mastering Atari with Discrete World Modelscited by 990
28 more
  1. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  2. 2016Generative Adversarial Text to Image Synthesiscited by 735
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2015Generating Sentences from a Continuous Spacecited by 607
  5. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  6. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  7. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  9. 2019Learning Latent Plans from Playcited by 310
  10. 2019Causal Confusion in Imitation Learningcited by 286
  11. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  12. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  13. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  14. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  15. 2020Transformers for One-Shot Visual Imitationcited by 77
  16. 2017Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructionscited by 61
  17. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  18. 2018Interactive Visual Grounding of Referring Expressions for Human-Robot Interactioncited by 55
  19. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
  20. 2018Mapping Navigation Instructions to Continuous Control Actions with Position-Visitation Predictioncited by 35
  21. 2021INVIGORATE: Interactive Visual Grounding and Grasping in Cluttercited by 32
  22. 2022Affordance Learning from Play for Sample-Efficient Policy Learningcited by 27
  23. 2020Robot Object Retrieval with Contextual Natural Language Queriescited by 26
  24. 2020Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learningcited by 22
  25. 2021Composing Pick-and-Place Tasks By Grounding Languagecited by 20
  26. 2019Adversarial Skill Networks: Unsupervised Robot Skill Learning from Videocited by 17
  27. 2020Spatial Reasoning from Natural Language Instructions for Robot Manipulationcited by 16
  28. 2021Hierarchical Policies for Cluttered-Scene Grasping with Latent Planscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 31; 2024: 38; 2025: 38; 2026: 18.

137 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  5. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  6. 2022Visual Language Maps for Robot Navigationcited by 362
  7. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  8. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  9. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  10. 20243D Diffuser Actor: Policy Diffusion with 3D Scene Representationscited by 278
  11. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  12. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
38 more
  1. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  2. 2024RT-H: Action Hierarchies Using Languagecited by 204
  3. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  4. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  5. 2025mimic-video: Video-Action Models for Generalizable Robot Control Beyond VLAscited by 133
  6. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  7. 2025Unified Vision-Language-Action Modelcited by 128
  8. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  9. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  10. 2024Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and Aviationcited by 108
  11. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 105
  12. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  13. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  14. 2025PD-VLA: Accelerating Vision-Language-Action Model Integrated with Action Chunking via Parallel Decodingcited by 88
  15. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  16. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  17. 2024The Ingredients for Robotic Diffusion Transformerscited by 72
  18. 2024Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulationcited by 71
  19. 2022VLMbench: A Compositional Benchmark for Vision-and-Language Manipulationcited by 68
  20. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 68
  21. 2024Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Roboticscited by 68
  22. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  23. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Modelscited by 58
  24. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 49
  25. 2023PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Playcited by 48
  26. 2025VLAS: Vision-Language-Action Model With Speech Instructions For Customized Robot Manipulationcited by 48
  27. 2024Efficient Diffusion Transformer Policies with Mixture of Expert Denoisers for Multitask Learningcited by 46
  28. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 46
  29. 2024VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulationcited by 45
  30. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 42
  31. 2024Diffusion Transformer Policycited by 41
  32. 2023KITE: Keypoint-Conditioned Policies for Semantic Manipulationcited by 39
  33. 2023Compositional Diffusion-Based Continuous Constraint Solverscited by 39
  34. 2024GR-MG: Leveraging Partially Annotated Data via Multi-Modal Goal-Conditioned Policycited by 39
  35. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  36. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  37. 2024Inference-Time Policy Steering through Human Interactionscited by 35
  38. 2024From LLMs to Actions: Latent Codes as Bridges in Hierarchical Robot Controlcited by 34

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics
  2. 2024Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning
  3. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  4. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  5. 2023Conditionally Combining Robot Skills using Large Language Models
  6. 2020Translating Natural Language Instructions to Computer Programs for Robot Manipulation
  7. 2024Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping
  8. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition