mlsanityData to 7 Oct 2026

Code as Reward: Empowering Reinforcement Learning with VLMs

arXiv 2402.04764 ↗First posted 7 Feb 2024cs.LGcited by 38

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Pre-trained Vision-Language Models (VLMs) are able to understand visual concepts, describe and decompose complex tasks into sub-tasks, and provide feedback on task completion. In this paper, we aim to leverage these capabilities to support the training of reinforcement learning (RL) agents. In principle, VLMs are well suited for this purpose, as they can naturally analyze image-based observations and provide feedback (reward) on learning progress. However, inference in VLMs is computationally expensive, so querying them frequently to compute rewards would significantly slowdown the training of an RL agent. To address this challenge, we propose a framework named Code as Reward (VLM-CaR). VLM-CaR produces dense reward functions from VLMs through code generation, thereby significantly reducing the computational burden of querying the VLM directly. We show that the dense rewards generated through our approach are very accurate across a diverse set of discrete and continuous environments, and can be more effective in training RL policies than the original sparse environment rewards.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 47 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

32 of 47 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Vision-Language Models as a Source of Rewards14 Dec
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  3. 2023Reinforcement Learning Friendly Vision-Language Model for Minecraft19 Mar
  4. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models21 Nov

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  7. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  9. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  10. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  11. 2023Learning Interactive Real-World Simulatorscited by 440
  12. 2016The Option-Critic Architecturecited by 433
16 more
  1. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  2. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  3. 2023Reward Design with Language Modelscited by 257
  4. 2023Video Language Planningcited by 160
  5. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  6. 2023Vision-Language Models as Success Detectorscited by 138
  7. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  8. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  9. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 57
  10. 2021panda-gym: Open-source goal-conditioned environments for robotic learningcited by 56
  11. 2019Dealing with Sparse Rewards in Reinforcement Learningcited by 40
  12. 2020What can I do here? A Theory of Affordances in Reinforcement Learningcited by 33
  13. 2020Options of Interest: Temporal Abstraction with Interest Functionscited by 30
  14. 2021Flexible Option Learningcited by 24
  15. 2018Simplifying Reward Design through Divide-and-Conquercited by 12
  16. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 15; 2026: 13.

38 papers.

Sort:

By area:

  1. 2024Video as the New Language for Real-World Decision Makingcited by 86
  2. 2024Vision Language Models are In-Context Value Learnerscited by 84
  3. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  4. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  5. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  6. 2025Self-Improving Embodied Foundation Modelscited by 28
  7. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  8. 2025Reinforcement Learning for Machine Learning Engineering Agentscited by 13
  9. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
  10. 2024Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racingcited by 7
  11. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Modelcited by 7
  12. 2025DYSTIL: Dynamic Strategy Induction with Large Language Models for Reinforcement Learningcited by 7
26 more
  1. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  2. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  3. 2025ReasonX: MLLM-Guided Intrinsic Image Decompositioncited by 5
  4. 2024Automated Rewards via LLM-Generated Progress Functionscited by 4
  5. 2025LuciBot: Automated Robot Policy Learning from Generated Videoscited by 4
  6. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  7. 2026Safe and Interpretable Multimodal Path Planning for Multi-Agent Cooperationcited by 3
  8. 2026Progress Reward Modeling for Robotic Learning: A Comprehensive Surveycited by 3
  9. 2025LLM-based Interactive Imitation Learning for Robotic Manipulationcited by 2
  10. 2025Leveraging LLMs for reward function design in reinforcement learning control taskscited by 2
  11. 2026MARVL: Multi-Stage Guidance for Robotic Manipulation via Vision-Language Modelscited by 2
  12. 2025Towards Automated Semantic Interpretability in Reinforcement Learning via Vision-Language Modelscited by 1
  13. 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learningcited by 1
  14. 2026From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Modelscited by 1
  15. 2025ROAD: Responsibility-Oriented Reward Design for Reinforcement Learning in Autonomous Drivingcited by 0
  16. 2025Which Programming Language and Model Work Best With LLM-as-a-Judge For Code Retrieval?cited by 0
  17. 2025GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learningcited by 0
  18. 2025BLIP-FusePPO: A Vision-Language Deep Reinforcement Learning Framework for Lane Keeping in Autonomous Vehiclescited by 0
  19. 2026TeNet: Text-to-Network for Compact Policy Synthesiscited by 0
  20. 2026Reward Shaping and Action Masking for Compositional Tasks using Behavior Trees and LLMscited by 0
  21. 2026AgenticRL: Agentic Reinforcement Learning with Self-Refinement for Complex UAV Navigationcited by 0
  22. 2026Learning Process Rewards via Success Visitation Matching for Efficient RLcited by 0
  23. 2026Weights or Skills? A Survey of Robot-Learning Techniques: from Action-Predicting Weights to Robots that Write their Own Skillscited by 0
  24. 2026Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teacherscited by 0
  25. 2026Same Trajectory, Contradictory Rewards (ROBORMBENCH): Paraphrase Fragility in Vision Language Reward Modelscited by 0
  26. 2026From Image Interpretation to Clinical Reasoning: Upstream Physician-Context-Aware Multimodal Learning with Causal Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  2. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  4. 2026Training a Conditioned Video Game Agent on a VLM Annotated Dataset
  5. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
  6. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning
  7. 2025Training-free Generation of Temporally Consistent Rewards from VLMs
  8. 2026Automating Potential-based Reward Shaping with Vision Language Model Guidance