mlsanityData to 7 Oct 2026

Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning

arXiv 2310.12921 ↗First posted 19 Oct 2023cs.LG cs.AIcited by 155

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) requires either manually specifying a reward function, which is often infeasible, or learning a reward model from a large amount of human feedback, which is often very expensive. We study a more sample-efficient alternative: using pretrained vision-language models (VLMs) as zero-shot reward models (RMs) to specify tasks via natural language. We propose a natural and general approach to using VLMs as reward models, which we call VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn complex tasks without a manually specified reward function, such as kneeling, doing the splits, and sitting in a lotus position. For each of these tasks, we only provide a single sentence text prompt describing the desired task with minimal prompt engineering. We provide videos of the trained agents at: https://sites.google.com/view/vlm-rm. We can improve performance by providing a second "baseline" prompt and projecting out parts of the CLIP embedding space irrelevant to distinguish between goal and baseline. Further, we find a strong scaling effect for VLM-RMs: larger VLMs trained with more compute and data are better reward models. The failure modes of VLM-RMs we encountered are all related to known capability limitations of current VLMs, such as limited spatial reasoning ability or visually unrealistic environments that are far off-distribution for the VLM. We find that VLM-RMs are remarkably robust as long as the VLM is large enough. This suggests that future VLMs will become more and more useful reward models for a wide range of RL applications.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 31 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

25 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents29 Jan

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  12. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
12 more
  1. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
  2. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  4. 2022Predictability and Surprise in Large Generative Modelscited by 209
  5. 2021Simple but Effective: CLIP Embeddings for Embodied AIcited by 191
  6. 2023Vision-Language Models as Success Detectorscited by 138
  7. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  8. 2023Training Language Models with Language Feedback at Scalecited by 125
  9. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  10. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  11. 2020Quantifying Differences in Reward Functionscited by 62
  12. 2021Cut the CARP: Fishing for zero-shot story evaluationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 48; 2025: 61; 2026: 45.

155 papers.

Sort:

By area:

  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 679
  2. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  3. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  4. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  5. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  6. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  7. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  8. 2024Vision Language Models are In-Context Value Learnerscited by 84
  9. 2023Vision-Language Models as a Source of Rewardscited by 65
  10. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  11. 2024Reasoning Paths with Reference Objects Elicit Quantitative Spatial Reasoning in Large Vision-Language Modelscited by 54
  12. 2024SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priorscited by 53
38 more
  1. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  2. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  3. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  4. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  5. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  6. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  7. 2024AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agentscited by 32
  8. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30
  9. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 29
  10. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
  11. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  12. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  13. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
  14. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 23
  15. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 20
  16. 2024A Survey of Optimization-based Task and Motion Planning: From Classical To Learning Approachescited by 19
  17. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 19
  18. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  19. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 18
  20. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 18
  21. 2024Natural Language Reinforcement Learningcited by 17
  22. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Roboticscited by 17
  23. 2025A Survey on Large Language Models for Automated Planningcited by 17
  24. 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 16
  25. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  26. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13
  27. 2025Truly Self-Improving Agents Require Intrinsic Metacognitive Learningcited by 13
  28. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 12
  29. 2024Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedbackcited by 10
  30. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  31. 2025Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learningcited by 10
  32. 2024Learning Manipulation Skills through Robot Chain-of-Thought with Sparse Failure Guidancecited by 9
  33. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
  34. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  35. 2025GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skillcited by 9
  36. 2025OpenGVL -- Benchmarking Visual Temporal Progress for Data Curationcited by 9
  37. 2024Adaptive Language-Guided Abstraction from Contrastive Explanationscited by 8
  38. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8

50 most cited of 155 All 155 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  2. 2023Language Reward Modulation for Pretraining Reinforcement Learning
  3. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models
  4. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning
  5. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  6. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  7. 2019From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
  8. 2026PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning