mlsanityData to 14 Oct 2026

FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning

arXiv 2406.00645 ↗First posted 2 Jun 2024cs.LG cs.AI cs.CVcited by 26

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

In this work, we investigate how to leverage pre-trained visual-language models (VLM) for online Reinforcement Learning (RL). In particular, we focus on sparse reward tasks with pre-defined textual task descriptions. We first identify the problem of reward misalignment when applying VLM as a reward in RL tasks. To address this issue, we introduce a lightweight fine-tuning method, named Fuzzy VLM reward-aided RL (FuRL), based on reward alignment and relay RL. Specifically, we enhance the performance of SAC/DrQ baseline agents on sparse reward tasks by fine-tuning VLM representations and using relay RL to avoid local minima. Extensive experiments on the Meta-world benchmark tasks demonstrate the efficacy of the proposed method. Code is available at: https://github.com/fuyw/FuRL.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 70 references that are in the map. First posted on arXiv 2 Jun 2024.

Builds on

58 of 70 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learning5 Feb
  2. 2023Vision-Language Models as a Source of Rewards14 Dec
  3. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  4. 2023Physically Grounded Vision-Language Models for Robotic Manipulation5 Sep
  5. 2023Distilling Internet-Scale Vision-Language Models into Embodied Agents29 Jan

Other references · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2015VQA: Visual Question Answeringcited by 2,506
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  11. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
38 more
  1. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  2. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  3. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 953
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  5. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  7. 2019Solving Rubik's Cube with a Robot Handcited by 734
  8. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  10. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  11. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  12. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  13. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  14. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  15. 2017Embodied Question Answeringcited by 336
  16. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  17. 2022Defining and Characterizing Reward Hackingcited by 203
  18. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  19. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
  20. 2020Adaptive Risk Minimization: Learning to Adapt to Domain Shiftcited by 183
  21. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  22. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 174
  23. 2019Active Domain Randomizationcited by 170
  24. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  25. 2023Vision-Language Models as Success Detectorscited by 142
  26. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  27. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  28. 2022Jump-Start Reinforcement Learningcited by 112
  29. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 103
  30. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  31. 2019Defense Against Adversarial Attacks Using Feature Scattering-based Adversarial Trainingcited by 95
  32. 2020Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexitycited by 92
  33. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 86
  34. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 67
  35. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 43
  36. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  37. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  38. 2022Generative Planning for Temporally Coordinated Exploration in Reinforcement Learningcited by 10

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 13; 2026: 6.

26 papers.

Sort:

By area:

  1. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 60
  2. 2024VLM-RL: A Unified Vision Language Models and Reinforcement Learning Framework for Safe Autonomous Drivingcited by 51
  3. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  4. 2025Iterative Tool Usage Exploration for Multimodal Agents via Step-wise Preference Tuningcited by 19
  5. 2024Robot Policy Learning with Temporal Optimal Transport Rewardcited by 8
  6. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  7. 2024The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewardscited by 7
  8. 2024Online Preference-based Reinforcement Learning with Self-augmented Feedback from Large Language Modelcited by 7
  9. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 7
  10. 2025Imitation Learning from a Single Temporally Misaligned Videocited by 7
  11. 2024YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controlscited by 5
  12. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
14 more
  1. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
  2. 2025Goal-Driven Reward by Video Diffusion Models for Reinforcement Learningcited by 3
  3. 2025Human-Aligned Procedural Level Generation Reinforcement Learning via Text-Level-Sketch Shared Representationcited by 2
  4. 2025LaGEA: Language Guided Embodied Agents for Robotic Manipulationcited by 2
  5. 2024SEAL: SEmantic-Augmented Imitation Learning via Language Modelcited by 1
  6. 2025Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learningcited by 1
  7. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimizationcited by 1
  8. 2025MATRIX: Multimodal Agent Tuning for Robust Tool-Use Reasoningcited by 1
  9. 2026COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Controlcited by 1
  10. 2026MVR: Multi-view Video Reward Shaping for Reinforcement Learningcited by 1
  11. 2026Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RLcited by 1
  12. 2024Fostering Intrinsic Motivation in Reinforcement Learning with Pretrained Foundation Modelscited by 0
  13. 2026VLM-Guided Experience Replaycited by 0
  14. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Code as Reward: Empowering Reinforcement Learning with VLMs
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
  3. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learning
  4. 2025Teaching RL Agents to Act Better: VLM as Action Advisor for Online Reinforcement Learning
  5. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  6. 2025The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning
  7. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
  8. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle