mlsanityData to 7 Oct 2026

Distilling Internet-Scale Vision-Language Models into Embodied Agents

arXiv 2301.12507 ↗First posted 29 Jan 2023cs.AIcited by 30

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Instruction-following agents must ground language into their observation and action spaces. Learning to ground language is challenging, typically requiring domain-specific engineering or large quantities of human interaction data. To address this challenge, we propose using pretrained vision-language models (VLMs) to supervise embodied agents. We combine ideas from model distillation and hindsight experience replay (HER), using a VLM to retroactively generate language describing the agent's behavior. Simple prompting allows us to control the supervision signal, teaching an agent to interact with novel objects based on their names (e.g., planes) or their features (e.g., colors) in a 3D rendered environment. Fewshot prompting lets us teach abstract category membership, including pre-existing categories (food vs toys) and ad-hoc ones (arbitrary preferences over objects). Our work outlines a new and effective way to use internet-scale VLMs, repurposing the generic language grounding acquired by such models to teach task-relevant groundings to embodied agents.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 81 references that are in the map. First posted on arXiv 29 Jan 2023.

Builds on

53 of 81 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov

Other references · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
38 more
  1. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  2. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  3. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  4. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  5. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  7. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  8. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  9. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  10. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  11. 2021iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Taskscited by 191
  12. 2022ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddingscited by 177
  13. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  14. 2022F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Modelscited by 149
  15. 2021Airbert: In-domain Pretraining for Vision-and-Language Navigationcited by 129
  16. 2021Skill Induction and Planning with Latent Languagecited by 99
  17. 2017Mapping Instructions and Visual Observations to Actions with Reinforcement Learningcited by 80
  18. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  19. 2019Robust Navigation with Language Pretraining and Stochastic Samplingcited by 77
  20. 2017Gated-Attention Architectures for Task-Oriented Language Groundingcited by 72
  21. 2020Imitating Interactive Intelligencecited by 55
  22. 2020Grounded Language Learning Fast and Slowcited by 54
  23. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  24. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  25. 2022Inferring Rewards from Language in Contextcited by 50
  26. 2022Learning from Unlabeled 3D Environments for Vision-and-Language Navigationcited by 49
  27. 2019Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flightcited by 43
  28. 2020Learning Rewards from Linguistic Feedbackcited by 41
  29. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40
  30. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  31. 2018Guiding Policies with Language via Meta-Learningcited by 34
  32. 2019RTFM: Generalising to Novel Environment Dynamics via Readingcited by 33
  33. 2020Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learningcited by 32
  34. 2021Interactive Learning from Activity Descriptioncited by 26
  35. 2018Interactive Grounded Language Acquisition and Generalization in a 2D Worldcited by 21
  36. 2020Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Followingcited by 18
  37. 2019ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learningcited by 17
  38. 2020Explore and Explain: Self-supervised Navigation and Recountingcited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 8; 2025: 7; 2026: 5.

30 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023Cognitive Architectures for Language Agentscited by 422
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  5. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  6. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  7. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  8. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  9. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  10. 2024Explore until Confident: Efficient Exploration for Embodied Question Answeringcited by 74
  11. 2023Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorldcited by 65
  12. 2023Vision-Language Models as a Source of Rewardscited by 65
18 more
  1. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
  2. 2023Towards A Unified Agent with Foundation Modelscited by 46
  3. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  4. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  5. 2024Object-Centric Instruction Augmentation for Robotic Manipulationcited by 16
  6. 2025GTR: Guided Thought Reinforcement Prevents Thought Collapse in RL-based VLM Agent Trainingcited by 14
  7. 2024Embodied CoT Distillation From LLM To Off-the-shelf Agentscited by 9
  8. 2025EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLMcited by 5
  9. 2025DeliveryBench: Can Agents Earn Profit in Real World?cited by 5
  10. 2025Multi-level Value Alignment in Agentic AI Systems: Survey and Perspectivescited by 4
  11. 2025GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Trainingcited by 3
  12. 2026Self-Supervised Bootstrapping of Action-Predictive Embodied Reasoningcited by 3
  13. 2025Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learningcited by 1
  14. 2025Self-Consistent Model-based Adaptation for Visual Reinforcement Learningcited by 0
  15. 2026Language-Critique Imitation Learning from Suboptimal Demonstrationscited by 0
  16. 2026Instance-Enriched Semantic Maps for Visual Language Navigationcited by 0
  17. 2026Belief Without Behavior: Measuring the Translation of Theory of Mind into Coordinated Social Action in Vision-Language Modelscited by 0
  18. 2026Act with Intent: Distilling Behavior Intent for Vision-Language-Action Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective
  2. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learning
  3. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning
  4. 2026EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents
  5. 2025From Grounding to Manipulation: Case Studies of Foundation Model Integration in Embodied Robotic Systems
  6. 2025Igniting VLMs toward the Embodied Space
  7. 2024VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought
  8. 2024HELPER-X: A Unified Instructable Embodied Agent to Tackle Four Interactive Vision-Language Domains with Memory-Augmented Language Models