mlsanityData to 14 Oct 2026

Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models

arXiv 2506.12822 ↗First posted 15 Jun 2025cs.LG cs.ROcited by 15

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Designing effective reward functions remains a fundamental challenge in reinforcement learning (RL), as it often requires extensive human effort and domain expertise. While RL from human feedback has been successful in aligning agents with human intent, acquiring high-quality feedback is costly and labor-intensive, limiting its scalability. Recent advancements in foundation models present a promising alternative--leveraging AI-generated feedback to reduce reliance on human supervision in reward learning. Building on this paradigm, we introduce ERL-VLM, an enhanced rating-based RL method that effectively learns reward functions from AI feedback. Unlike prior methods that rely on pairwise comparisons, ERL-VLM queries large vision-language models (VLMs) for absolute ratings of individual trajectories, enabling more expressive feedback and improved sample efficiency. Additionally, we propose key enhancements to rating-based RL, addressing instability issues caused by data imbalance and noisy labels. Through extensive experiments across both low-level and high-level control tasks, we demonstrate that ERL-VLM significantly outperforms existing VLM-based reward generation methods. Our results demonstrate the potential of AI feedback for scaling RL with minimal human intervention, paving the way for more autonomous and efficient reward learning.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 73 references that are in the map. First posted on arXiv 15 Jun 2025.

Builds on

53 of 73 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Code as Reward: Empowering Reinforcement Learning with VLMs7 Feb
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback6 Feb
  3. 2023Vision-Language Models as a Source of Rewards14 Dec
  4. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  5. 2023Reinforcement Learning Friendly Vision-Language Model for Minecraft19 Mar

Other references · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2016Generative Adversarial Imitation Learningcited by 1,597
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  9. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  10. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  11. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
36 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  4. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  6. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  8. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  9. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  10. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  11. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  13. 2023Reward Design with Language Modelscited by 261
  14. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  15. 2022Is GPT-3 a Good Data Annotator?cited by 208
  16. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
  17. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  18. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  19. 2022Mastering the Game of Stratego with Model-Free Multiagent Reinforcement Learningcited by 131
  20. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  21. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  22. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  23. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  24. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  25. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  26. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 87
  27. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  28. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
  29. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  30. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  31. 2024Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviorscited by 25
  32. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  33. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 21
  34. 2023Rating-based Reinforcement Learningcited by 15
  35. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrationscited by 7
  36. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
  2. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learningcited by 11
  3. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  4. 2025Occlusion-robust Stylization for Drawing-based 3D Animationcited by 3
  5. 2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learningcited by 3
  6. 2026RDA: Reward Design Agent for Reinforcement Learningcited by 3
  7. 2026Automating Potential-based Reward Shaping with Vision Language Model Guidancecited by 2
  8. 2025Teaching RL Agents to Act Better: VLM as Action Advisor for Online Reinforcement Learningcited by 1
  9. 2026AirSplat: Alignment and Rating for Robust Feed-Forward 3D Gaussian Splattingcited by 1
  10. 2026Learning Process Rewards via Success Visitation Matching for Efficient RLcited by 1
  11. 2025Oracle-RLAIF: An Improved Fine-Tuning Framework for Multi-modal Video Models using Reinforcement Learning from Ranking Feedbackcited by 0
  12. 2026Off-Policy Actor-Critic with Sigmoid-Bounded Entropy for Real-World Robot Learningcited by 0
3 more
  1. 2026VLM-Guided Experience Replaycited by 0
  2. 2026IVR-R1: Refining Trajectories through Iterative Visual-Grounded Reasoning in Reinforcement Learningcited by 0
  3. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
  2. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences
  5. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning
  6. 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning
  7. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  8. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback