mlsanityData to 7 Oct 2026

Vision-Language Models Provide Promptable Representations for Reinforcement Learning

arXiv 2402.02651 ↗First posted 5 Feb 2024cs.LG cs.AI cs.CVcited by 41

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Humans can quickly learn new behaviors by leveraging background world knowledge. In contrast, agents trained with reinforcement learning (RL) typically learn behaviors from scratch. We thus propose a novel approach that uses the vast amounts of general and indexable world knowledge encoded in vision-language models (VLMs) pre-trained on Internet-scale data for embodied RL. We initialize policies with VLMs by using them as promptable representations: embeddings that encode semantic features of visual observations based on the VLM's internal knowledge and reasoning capabilities, as elicited through prompts that provide task context and auxiliary information. We evaluate our approach on visually-complex, long horizon RL tasks in Minecraft and robot navigation in Habitat. We find that our policies trained on embeddings from off-the-shelf, general-purpose VLMs outperform equivalent policies trained on generic, non-promptable image embeddings. We also find our approach outperforms instruction-following methods and performs comparably to domain-specific embeddings. Finally, we show that our approach can use chain-of-thought prompting to produce representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 95 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

51 of 95 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  2. 2023Towards A Unified Agent with Foundation Models18 Jul
  3. 2022Visual Language Maps for Robot Navigation11 Oct

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
36 more
  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  7. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  8. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  9. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  10. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
  11. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  12. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  13. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  14. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  15. 2023Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memorycited by 288
  16. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  17. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  18. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  19. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  20. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  21. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  22. 2022Habitat-Web: Learning Embodied Object-Search Strategies from Human Demonstrations at Scalecited by 116
  23. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  24. 2021Skill Induction and Planning with Latent Languagecited by 99
  25. 2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 95
  26. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  27. 2022Correcting Robot Plans with Natural Language Feedbackcited by 68
  28. 2023OVRL-V2: A simple state-of-art baseline for ImageNav and ObjectNavcited by 68
  29. 2023Learning to Model the World with Languagecited by 66
  30. 2019Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddingscited by 55
  31. 2023Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Predictioncited by 39
  32. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  33. 2022Stubborn: A Strong Baseline for Indoor Object Navigationcited by 31
  34. 2022Affordance Learning from Play for Sample-Efficient Policy Learningcited by 27
  35. 2023LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architecturescited by 27
  36. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 22; 2026: 7.

41 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  3. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  4. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  5. 2025Improving Vision-Language-Action Model with Online Reinforcement Learningcited by 107
  6. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  7. 2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
  8. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  9. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  10. 2024MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videoscited by 43
  11. 2025SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectoriescited by 29
  12. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
29 more
  1. 2025Digi-Q: Learning Q-Value Functions for Training Device-Control Agentscited by 20
  2. 2024Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitationcited by 18
  3. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  4. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  5. 2024KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Datacited by 15
  6. 2025GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networkscited by 15
  7. 2024Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Modelscited by 14
  8. 2025CL-CoTNav: Closed-Loop Hierarchical Chain-of-Thought for Zero-Shot Object-Goal Navigation with Vision-Language Modelscited by 11
  9. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  10. 2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
  11. 2026Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RLcited by 5
  12. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3
  13. 2025VLM Q-Learning: Aligning Vision-Language Models for Interactive Decision-Makingcited by 3
  14. 2025Sample Efficient Reinforcement Learning via Large Vision Language Model Distillationcited by 3
  15. 2025GoalLadder: Incremental Goal Discovery with Vision-Language Modelscited by 3
  16. 2025Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamicscited by 2
  17. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2
  18. 2025Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learningcited by 1
  19. 2025Towards Automated Semantic Interpretability in Reinforcement Learning via Vision-Language Modelscited by 1
  20. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  21. 2025Interpret Policies in Deep Reinforcement Learning using SILVER with RL-Guided Labeling: A Model-level Approach to High-dimensional and Multi-action Environmentscited by 1
  22. 2026Vision-Language Models Unlock Task-Centric Latent Actionscited by 1
  23. 2025Curriculum-Based Multi-Tier Semantic Exploration via Deep Reinforcement Learningcited by 0
  24. 2025Medical Vision Language Models as Policies for Robotic Surgerycited by 0
  25. 2025Recurrent Off-Policy Deep Reinforcement Learning Doesn't Have to be Slowcited by 0
  26. 2026MVR: Multi-view Video Reward Shaping for Reinforcement Learningcited by 0
  27. 2026Chain-of-Adaptation: Surgical Vision-Language Adaptation with Reinforcement Learningcited by 0
  28. 2026Act on What You See: Unlocking Safe Social Navigation in Vision-Language-Action Modelscited by 0
  29. 2026Prompt-Driven Exploration: Language as an Exploration Space for VLA Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting
  2. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success
  3. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
  4. 2025Can Pretrained Vision-Language Embeddings Alone Guide Robot Navigation?
  5. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents
  6. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  7. 2026Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Models
  8. 2024From Goal-Conditioned to Language-Conditioned Agents via Vision-Language Models