mlsanityData to 7 Oct 2026

Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning

arXiv 2502.01616 ↗First posted 3 Feb 2025cs.LGcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based reinforcement learning (RL) offers a promising approach for aligning policies with human intent but is often constrained by the high cost of human feedback. In this work, we introduce PrefVLM, a framework that integrates Vision-Language Models (VLMs) with selective human feedback to significantly reduce annotation requirements while maintaining performance. Our method leverages VLMs to generate initial preference labels, which are then filtered to identify uncertain cases for targeted human annotation. Additionally, we adapt VLMs using a self-supervised inverse dynamics loss to improve alignment with evolving policies. Experiments on Meta-World manipulation tasks demonstrate that PrefVLM achieves comparable or superior success rates to state-of-the-art methods while using up to 2 x fewer human annotations. Furthermore, we show that adapted VLMs enable efficient knowledge transfer across tasks, further minimizing feedback needs. Our results highlight the potential of combining VLMs with selective human supervision to make preference-based RL more scalable and practical.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 48 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

30 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Models19 Oct
  2. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning18 Mar
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  9. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  10. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  11. 2020Does label smoothing mitigate label noise?cited by 204
  12. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
12 more
  1. 2019Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networkscited by 158
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  3. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  4. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  5. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  6. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  7. 2020Denoising Implicit Feedback for Recommendationcited by 80
  8. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  9. 2021Cross-domain Imitation from Observationscited by 42
  10. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 42
  11. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  12. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Generative AI for Autonomous Driving: Frontiers and Opportunitiescited by 7
  2. 2025Human Implicit Preference-Based Policy Fine-tuning for Multi-Agent Reinforcement Learning in USV Swarmcited by 5
  3. 2026Digital Twin AI: Opportunities and Challenges from Large Language Models to World Modelscited by 4
  4. 2026VLM-Guided Experience Replaycited by 0
  5. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learningcited by 0
  6. 2026CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  2. 2024PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models ↗
  3. 2026Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning
  4. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  6. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
  7. 2026Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback
  8. 2025Test-Time Reasoning Through Visual Human Preferences with VLMs and Soft Rewards