mlsanityData to 7 Oct 2026

Vision-Language Models as a Source of Rewards

arXiv 2312.09187 ↗First posted 14 Dec 2023cs.LGcited by 65

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for building generalist agents with RL has been the need for a large number of reward functions for achieving different goals. We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of rewards for reinforcement learning agents. We show how rewards for visual achievement of a variety of language goals can be derived from the CLIP family of models, and used to train RL agents that can achieve a variety of language goals. We showcase this approach in two distinct visual domains and present a scaling trend showing how larger VLMs lead to more accurate rewards for visual goal achievement, which in turn produces more capable RL agents.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 43 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

32 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  2. 2023Reinforcement Learning Friendly Vision-Language Model for Minecraft19 Mar
  3. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents29 Jan
  4. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  7. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  8. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  9. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  10. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  12. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
16 more
  1. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  2. 2023Reward Design with Language Modelscited by 257
  3. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  4. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  5. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  6. 2023Vision-Language Models as Success Detectorscited by 138
  7. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  8. 2021Self-Supervised Learning with Swin Transformerscited by 121
  9. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  10. 2023Video Prediction Models as Rewards for Reinforcement Learningcited by 76
  11. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  12. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  13. 2023Language Instructed Reinforcement Learning for Human-AI Coordinationcited by 61
  14. 2021Muesli: Combining Improvements in Policy Optimizationcited by 53
  15. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  16. 2021Creating Multimodal Interactive Agents with Imitation and Self-Supervised Learningcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 18; 2026: 20.

65 papers.

Sort:

By area:

  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  2. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  3. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  4. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequencescited by 93
  5. 2024Video as the New Language for Real-World Decision Makingcited by 86
  6. 2024Vision Language Models are In-Context Value Learnerscited by 84
  7. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  8. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  9. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  10. 2024Code as Reward: Empowering Reinforcement Learning with VLMscited by 38
  11. 2024GenRL: Multimodal-foundation world models for generalization in embodied agentscited by 29
  12. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 29
38 more
  1. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  2. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  3. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
  4. 2024Continuously Improving Mobile Manipulation with Autonomous Real-World RLcited by 20
  5. 2024BlenderAlchemy: Editing 3D Graphics with Vision-Language Modelscited by 18
  6. 2024Video-Language Critic: Transferable Reward Functions for Language-Conditioned Roboticscited by 17
  7. 2025A Survey on Large Language Models for Automated Planningcited by 17
  8. 2026BPP: Long-Context Robot Imitation Learning by Focusing on Key History Framescited by 17
  9. 2025AppVLM: A Lightweight Vision Language Model for Online App Controlcited by 15
  10. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  11. 2024Learning Manipulation Skills through Robot Chain-of-Thought with Sparse Failure Guidancecited by 9
  12. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  13. 2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
  14. 2025FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Makingcited by 8
  15. 2024Assessing the Zero-Shot Capabilities of LLMs for Action Evaluation in RLcited by 7
  16. 2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7
  17. 2024RLZero: Direct Policy Inference from Language Without In-Domain Supervisioncited by 7
  18. 2025Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Actingcited by 6
  19. 2025ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Taskscited by 6
  20. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  21. 2024YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controlscited by 5
  22. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
  23. 2024MotIF: Motion Instruction Fine-tuningcited by 4
  24. 2025RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skillscited by 4
  25. 2026A Progressive Training Strategy for Embodied Vision-Language Models to Mitigate Spatio-Temporal Hallucinationscited by 4
  26. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3
  27. 2024LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemblecited by 3
  28. 2025DriveMind: A Dual Visual Language Model-based Reinforcement Learning Framework for Autonomous Drivingcited by 3
  29. 2025GoalLadder: Incremental Goal Discovery with Vision-Language Modelscited by 3
  30. 2026DriveVLM-RL: Neuroscience-Inspired Reinforcement Learning with Vision-Language Models for Safe and Deployable Autonomous Drivingcited by 3
  31. 2026Progress Reward Modeling for Robotic Learning: A Comprehensive Surveycited by 3
  32. 2025Offline RLAIF: Piloting VLM Feedback for RL via SFOcited by 2
  33. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulationcited by 2
  34. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2
  35. 2025Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learningcited by 1
  36. 2025VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision-Language Modelscited by 1
  37. 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learningcited by 1
  38. 2026RDA: Reward Design Agent for Reinforcement Learningcited by 1

50 most cited of 65 All 65 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  3. 2024Visual Grounding for Object-Level Generalization in Reinforcement Learning
  4. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
  5. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agents
  6. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Success
  7. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning
  8. 2025G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning