mlsanityData to 14 Oct 2026

CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning

arXiv 2607.01721 ↗First posted 2 Jul 2026cs.ROcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Reward design remains a central challenge in reinforcement learning (RL). Hand-crafted rewards are often difficult to specify and may lead to suboptimal policies, while learned rewards from preferences can suffer from inefficiency and unstable training. Inspired by the dual nature of human learning explored in cognitive science, we decompose rewards into two complementary components: Formal Rewards (FR), explicitly designed based on task knowledge, and Residual Rewards (RR), learned from observations to capture implicit and nuanced preferences. Based on this decomposition, we propose CoRe, a hybrid framework that integrates FR and RR with vision-language models (VLMs) feedback to achieve preference-aligned policies without human involvement. Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design and optimize FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capturing nuanced rewards that complement FR, ensuring alignment with human intent. Through the synergy of FRM and RRM, CoRe enables the automatic construction of reliable rewards that are efficient and preference-aligned. Extensive experiments demonstrate that CoRe outperforms existing approaches in terms of policy learning effectiveness and efficiency on ten robotic manipulation tasks in simulation and five real-worlds. Videos can be found on our project website: https://core-2026.github.io/

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 69 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

52 of 69 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Enhancing Rating-Based Reinforcement Learning to Effectively Leverage Feedback from Large Vision-Language Models15 Jun
  2. 2024ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Robotics27 Nov
  3. 2024Real-World Offline Reinforcement Learning from Vision Language Model Feedback8 Nov
  4. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback6 Feb
  5. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning19 Oct

Other references · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  9. 2016Concrete Problems in AI Safetycited by 1,642
  10. 2016Generative Adversarial Imitation Learningcited by 1,597
  11. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
35 more
  1. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  2. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  4. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  5. 2023Language to Rewards for Robotic Skill Synthesiscited by 312
  6. 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 311
  7. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  8. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 225
  9. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 188
  10. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 134
  11. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 121
  12. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  13. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 102
  14. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  15. 2022Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation?cited by 70
  16. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  17. 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 61
  18. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 57
  19. 2023Language Reward Modulation for Pretraining Reinforcement Learningcited by 43
  20. 2024RIME: Robust Preference-based Reinforcement Learning with Noisy Preferencescited by 39
  21. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 26
  22. 2024CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Modelscited by 22
  23. 2023Efficient Preference-Based Reinforcement Learning Using Learned Dynamics Modelscited by 21
  24. 2024Learning Reward for Robot Skills Using Large Language Models via Self-Alignmentcited by 20
  25. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learningcited by 15
  26. 2023Rating-based Reinforcement Learningcited by 15
  27. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  28. 2024Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learningcited by 9
  29. 2025VLP: Vision-Language Preference Learning for Embodied Manipulationcited by 8
  30. 2024Navigating Noisy Feedback: Enhancing Reinforcement Learning with Error-Prone Language Modelscited by 7
  31. 2025Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learningcited by 7
  32. 2024Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolutioncited by 5
  33. 2024SDS -- See it, Do it, Sorted: Quadruped Skill Synthesis from Single Video Demonstrationcited by 4
  34. 2023Data Driven Reward Initialization for Preference based Reinforcement Learningcited by 3
  35. 2024Language-Model-Assisted Bi-Level Programming for Reward Learning from Internet Videoscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  2. 2026VLM-AR3L: Vision-Language Models for Absolute and Relative Rewards in Reinforcement Learning
  3. 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuning
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  5. 2025Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
  6. 2026Beyond Pairwise Feedback: Listwise Vision-Language Supervision for Preference-Based Reward Learning
  7. 2025VARP: Reinforcement Learning from Vision-Language Model Feedback with Agent Regularized Preferences
  8. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models