mlsanityData to 14 Oct 2026

Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning

arXiv 2504.08772 ↗First posted 3 Apr 2025cs.LG cs.AIcited by 2

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 2 more
Abstract (arXiv)

In offline reinforcement learning (RL), learning from fixed datasets presents a promising solution for domains where real-time interaction with the environment is expensive or risky. However, designing dense reward signals for offline dataset requires significant human effort and domain expertise. Reinforcement learning with human feedback (RLHF) has emerged as an alternative, but it remains costly due to the human-in-the-loop process, prompting interest in automated reward generation models. To address this, we propose Reward Generation via Large Vision-Language Models (RG-VLM), which leverages the reasoning capabilities of LVLMs to generate rewards from offline data without human involvement. RG-VLM improves generalization in long-horizon tasks and can be seamlessly integrated with the sparse reward signals to enhance task performance, demonstrating its potential as an auxiliary reward signal.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 37 references that are in the map. First posted on arXiv 3 Apr 2025.

Builds on

33 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  2. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  10. 2017Hindsight Experience Replaycited by 1,136
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  12. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
19 more
  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  3. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  5. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  6. 2017Inverse Reward Designcited by 249
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  8. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  9. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  10. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  11. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  12. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 66
  13. 2023Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RLcited by 36
  14. 2023Reinforcement Learning from Diverse Human Preferencescited by 24
  15. 2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
  16. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  17. 2023Primitive Skill-based Robot Learning from Human Evaluative Feedbackcited by 15
  18. 2021Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environmentcited by 7
  19. 2024Predictive Coding for Decision Transformercited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
  2. 2026VERTIGO: Visual Preference Optimization for Cinematic Camera Trajectory Generationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  2. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedback
  3. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  4. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  5. 2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning
  6. 2023Vision-Language Models as a Source of Rewards
  7. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
  8. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order