mlsanityData to 7 Oct 2026

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback

arXiv 2402.03681 ↗First posted 6 Feb 2024cs.RO cs.AI cs.LGcited by 147

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Reward engineering has long been a challenge in Reinforcement Learning (RL) research, as it often requires extensive human effort and iterative processes of trial-and-error to design effective reward functions. In this paper, we propose RL-VLM-F, a method that automatically generates reward functions for agents to learn new tasks, using only a text description of the task goal and the agent's visual observations, by leveraging feedbacks from vision language foundation models (VLMs). The key to our approach is to query these models to give preferences over pairs of the agent's image observations based on the text description of the task goal, and then learn a reward function from the preference labels, rather than directly prompting these models to output a raw reward score, which can be noisy and inconsistent. We demonstrate that RL-VLM-F successfully produces effective rewards and policies across various domains - including classic control, as well as manipulation of rigid, articulated, and deformable objects - without the need for human supervision, outperforming prior methods that use large pretrained models for reward generation under the same assumptions. Videos can be found on our project website: https://rlvlmf2024.github.io/

Key facts. Cited by 147 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 54 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

38 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct

Other references · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2016OpenAI Gymcited by 1,684
  8. 2016Generative Adversarial Imitation Learningcited by 1,576
  9. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  11. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  12. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
25 more
  1. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  4. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  5. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  6. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  7. 2023Reward Design with Language Modelscited by 257
  8. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  9. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  10. 2022Human-to-Robot Imitation in the Wildcited by 157
  11. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  12. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 118
  13. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  14. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  15. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modellingcited by 88
  16. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  17. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 70
  18. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  19. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  20. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matchingcited by 67
  21. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  22. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
  23. 2023Towards Generalizable Zero-Shot Manipulation via Translating Human Interaction Planscited by 51
  24. 2021Learning Multimodal Rewards from Rankingscited by 41
  25. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 29; 2025: 58; 2026: 60.

147 papers.

Sort:

By area:

  1. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  2. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  3. 2024Vision Language Models are In-Context Value Learnerscited by 84
  4. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  5. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 66
  6. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  7. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 49
  8. 2025World-in-World: World Models in a Closed-Loop Worldcited by 44
  9. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  10. 2024RL-GPT: Integrating Reinforcement Learning and Code-as-policycited by 32
  11. 2025SIMA 2: A Generalist Embodied Agent for Virtual Worldscited by 26
  12. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
38 more
  1. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
  2. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 20
  3. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedbackcited by 20
  4. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  5. 2024SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generationcited by 15
  6. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Modelscited by 15
  7. 2025FDPP: Fine-tune Diffusion Policy with Human Preferencecited by 14
  8. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Datacited by 13
  9. 2026PROGRESSLM: Towards Progress Reasoning in Vision-Language Modelscited by 13
  10. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 12
  11. 2026Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Modelscited by 11
  12. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learningcited by 10
  13. 2024Learning Manipulation Skills through Robot Chain-of-Thought with Sparse Failure Guidancecited by 9
  14. 2024On the Modeling Capabilities of Large Language Models for Sequential Decision Makingcited by 9
  15. 2025SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Modelscited by 9
  16. 2025LERO: LLM-driven Evolutionary framework with Hybrid Rewards and Enhanced Observation for Multi-Agent Reinforcement Learningcited by 9
  17. 2024Batch Active Learning of Reward Functions from Human Preferencescited by 8
  18. 2024ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Roboticscited by 8
  19. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  20. 2025Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Modelscited by 8
  21. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferencescited by 8
  22. 2025Reflection-Based Task Adaptation for Self-Improving VLAcited by 8
  23. 2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7
  24. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Modelscited by 7
  25. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Modelcited by 7
  26. 2025TREND: Tri-teaching for Robust Preference-based Reinforcement Learning with Demonstrationscited by 7
  27. 2025TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learningcited by 7
  28. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  29. 2024MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimizationcited by 6
  30. 2024SELU: Self-Learning Embodied MLLMs in Unknown Environmentscited by 6
  31. 2024MAPLE: A Framework for Active Preference Learning Guided by Large Language Modelscited by 6
  32. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 6
  33. 2025ROVER: Recursive Reasoning Over Videos with Vision-Language Models for Embodied Taskscited by 6
  34. 2026Position: Human-Centric AI Requires a Minimum Viable Level of Human Understandingcited by 6
  35. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  36. 2024YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controlscited by 5
  37. 2024CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learningcited by 5
  38. 2026GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planningcited by 5

50 most cited of 147 All 147 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  2. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning
  3. 2023Vision-Language Models as a Source of Rewards
  4. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  5. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  6. 2026Training a Conditioned Video Game Agent on a VLM Annotated Dataset
  7. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Models
  8. 2025Training-free Generation of Temporally Consistent Rewards from VLMs