mlsanityData to 14 Oct 2026

Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models

arXiv 2211.11736 ↗First posted 21 Nov 2022cs.RO cs.AI cs.LGcited by 58

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

In recent years, much progress has been made in learning robotic manipulation policies that follow natural language instructions. Such methods typically learn from corpora of robot-language data that was either collected with specific tasks in mind or expensively re-labelled by humans with rich language descriptions in hindsight. Recently, large-scale pretrained vision-language models (VLMs) like CLIP or ViLD have been applied to robotics for learning representations and scene descriptors. Can these pretrained models serve as automatic labelers for robot data, effectively importing Internet-scale knowledge into existing datasets to make them useful even for tasks that are not reflected in their ground truth annotations? To accomplish this, we introduce Data-driven Instruction Augmentation for Language-conditioned control (DIAL): we utilize semi-supervised language labels leveraging the semantic understanding of CLIP to propagate knowledge onto large datasets of unlabelled demonstration data and then train language-conditioned policies on the augmented datasets. This method enables cheaper acquisition of useful language descriptions compared to expensive human labels, allowing for more efficient label coverage of large-scale datasets. We apply DIAL to a challenging real-world robotic manipulation domain where 96.5% of the 80,000 demonstrations do not contain crowd-sourced language annotations. DIAL enables imitation learning policies to acquire new capabilities and generalize to 60 novel instructions unseen in the original dataset.

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 56 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

42 of 56 references.

References · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  9. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
30 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  2. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  3. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  5. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  7. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 523
  8. 2018Universal Sentence Encodercited by 489
  9. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  10. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  11. 2019Learning Latent Plans from Playcited by 316
  12. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  13. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 229
  14. 2022Interactive Language: Talking to Robots in Real Timecited by 218
  15. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 209
  16. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  17. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  18. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 138
  19. 2020Language-Conditioned Imitation Learning for Robot Manipulation Taskscited by 136
  20. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  21. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 118
  22. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  23. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  24. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  25. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Followingcited by 67
  26. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  27. 2020Thinking While Moving: Deep Reinforcement Learning with Concurrent Controlcited by 31
  28. 2019ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learningcited by 17
  29. 2019HIGhER : Improving instruction following with Hindsight Generation for Experience Replaycited by 13
  30. 2022Grounding Hindsight Instructions in Multi-Goal Reinforcement Learning for Roboticscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 15; 2024: 26; 2025: 10; 2026: 6.

58 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  4. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  5. 2023Augmented Language Models: a Surveycited by 403
  6. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  7. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  8. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  9. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  10. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  11. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 134
  12. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
38 more
  1. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
  2. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  3. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  4. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 74
  5. 2024Real-Time Anomaly Detection and Reactive Planning with Large Language Modelscited by 68
  6. 2023Vision-Language Models as a Source of Rewardscited by 66
  7. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 61
  8. 2023Language Models as Zero-Shot Trajectory Generatorscited by 60
  9. 2023Towards A Unified Agent with Foundation Modelscited by 47
  10. 2024Diffusion Transformer Policycited by 41
  11. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 39
  12. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  13. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  14. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  15. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agentscited by 30
  16. 2023Augmenting Autotelic Agents with Large Language Modelscited by 30
  17. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representationscited by 25
  18. 2024A Survey on Game Playing Agents and Large Models: Methods, Applications, and Challengescited by 22
  19. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Roboticscited by 19
  20. 2023Contrastive Language, Action, and State Pre-training for Robot Learningcited by 16
  21. 2024Object-Centric Instruction Augmentation for Robotic Manipulationcited by 16
  22. 2024Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Modelscited by 16
  23. 2024MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?cited by 15
  24. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  25. 2024STEER: Flexible Robotic Manipulation via Dense Language Groundingcited by 14
  26. 2022Policy Adaptation from Foundation Model Feedbackcited by 10
  27. 2023Benchmarks for Physical Reasoning AIcited by 8
  28. 2024VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actionscited by 8
  29. 2025Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigationcited by 8
  30. 2024Robo-MUTUAL: Robotic Multimodal Task Specification via Unimodal Learningcited by 6
  31. 2025CrayonRobo: Object-Centric Prompt-Driven Vision-Language-Action Model for Robotic Manipulationcited by 6
  32. 2024MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Modelscited by 5
  33. 2024Diffusion Augmented Agents: A Framework for Efficient Exploration and Transfer Learningcited by 5
  34. 2026FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policiescited by 5
  35. 2025EmbodiedCoder: Parameterized Embodied Mobile Manipulation via Modern Coding Modelcited by 3
  36. 2026Task Robustness via Re-Labelling Vision-Action Robot Datacited by 2
  37. 2024Temporal and Semantic Evaluation Metrics for Foundation Models in Post-Hoc Analysis of Robotic Sub-taskscited by 1
  38. 2024The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Datacited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  2. 2023Language-Driven Representation Learning for Robotics
  3. 2023Programmatically Grounded, Compositionally Generalizable Robotic Manipulation
  4. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
  5. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policy
  6. 2025CAST: Counterfactual Labels Improve Instruction Following in Vision-Language-Action Models
  7. 2026Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Models
  8. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control