mlsanityData to 7 Oct 2026

Reducing Oracle Feedback with Vision-Language Embeddings for Preference-Based RL

arXiv 2603.28053 ↗First posted 30 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Preference-based reinforcement learning can learn effective reward functions from comparisons, but its scalability is constrained by the high cost of oracle feedback. Lightweight vision-language embedding (VLE) models provide a cheaper alternative, but their noisy outputs limit their effectiveness as standalone reward generators. To address this challenge, we propose ROVED, a hybrid framework that combines VLE-based supervision with targeted oracle feedback. Our method uses the VLE to generate segment-level preferences and defers to an oracle only for samples with high uncertainty, identified through a filtering mechanism. In addition, we introduce a parameter-efficient fine-tuning method that adapts the VLE with the obtained oracle feedback in order to improve the model over time in a synergistic fashion. This ensures the retention of the scalability of embeddings and the accuracy of oracles, while avoiding their inefficiencies. Across multiple robotic manipulation tasks, ROVED matches or surpasses prior preference-based methods while reducing oracle queries by up to 80%. Remarkably, the adapted VLE generalizes across tasks, yielding cumulative annotation savings of up to 90%, highlighting the practicality of combining scalable embeddings with precise oracle supervision for preference-based RL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 28 of its 33 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

28 of 33 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  10. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  11. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
16 more
  1. 2020Does label smoothing mitigate label noise?cited by 204
  2. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  3. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  4. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
  5. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  6. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  7. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  8. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  9. 2020Denoising Implicit Feedback for Recommendationcited by 80
  10. 2018Learning Navigation Behaviors End-to-End with AutoRLcited by 70
  11. 2024A Survey on Deep Active Learning: Recent Advances and New Frontierscited by 53
  12. 2021Cross-domain Imitation from Observationscited by 42
  13. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 37
  14. 2024FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learningcited by 26
  15. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
  16. 2024Robust Offline Imitation Learning from Diverse Auxiliary Datacited by 4

Built on by

Not cited yet.