mlsanityData to 14 Oct 2026

Generative RLHF-V: Learning Principles from Multi-modal Human Preference

arXiv 2505.18531 ↗First posted 24 May 2025cs.AI cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Training multi-modal large language models (MLLMs) that align with human intentions is a long-term challenge. Traditional score-only reward models for alignment suffer from low accuracy, weak generalization, and poor interpretability, blocking the progress of alignment methods, e.g., reinforcement learning from human feedback (RLHF). Generative reward models (GRMs) leverage MLLMs' intrinsic reasoning capabilities to discriminate pair-wise responses, but their pair-wise paradigm makes it hard to generalize to learnable rewards. We introduce Generative RLHF-V, a novel alignment framework that integrates GRMs with multi-modal RLHF. We propose a two-stage pipeline: multi-modal generative reward modeling from RL, where RL guides GRMs to actively capture human intention, then predict the correct pair-wise scores; and RL optimization from grouped comparison, which enhances multi-modal RL scoring precision by grouped responses comparison. Experimental results demonstrate that, besides out-of-distribution generalization of RM discrimination, our framework improves 4 MLLMs' performance across 7 benchmarks by 18.1%, while the baseline RLHF is only 5.3%. We further validate that Generative RLHF-V achieves a near-linear improvement with an increasing number of candidate responses. Our code and models can be found at https://generative-rlhf-v.github.io.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 59 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

49 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr

Other references · 48

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  12. 2016Concrete Problems in AI Safetycited by 1,642
36 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  6. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  8. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  9. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  11. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 461
  12. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  13. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
  14. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  15. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  16. 2023AI Alignment: A Comprehensive Surveycited by 263
  17. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  18. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  19. 2024Generative Reward Modelscited by 148
  20. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  21. 2024Critique-out-Loud Reward Modelscited by 106
  22. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  23. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 83
  24. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  25. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 72
  26. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 66
  27. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  28. 2024Multimodal Situational Safetycited by 58
  29. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  30. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilitiescited by 57
  31. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  32. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  33. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 32
  34. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimizationcited by 21
  35. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Designcited by 15
  36. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  2. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  3. 2025SaFeR-VLM: Toward Safety-aware Fine-grained Reasoning in Multimodal Modelscited by 10
  4. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 6
  5. 2026RetouchIQ: MLLM Agents for Instruction-Based Image Retouching with Generalist Rewardcited by 4
  6. 2026Wan-R1: Verifiable-Reinforcement Learning for Video Reasoningcited by 4
  7. 2026Multimodal Reward Hacking in Reinforcement Learningcited by 2
  8. 2026SaFeR-Steer: Evolving Multi-Turn MLLMs via Synthetic Bootstrapping and Feedback Dynamicscited by 1
  9. 2026Towards Multi-modal Multi-turn Safety: From Agentic Interaction to Strategic Alignmentcited by 0
  10. 2026SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safetycited by 0
  11. 2026OOD-MMSafe: Advancing MLLM Safety from Harmful Intent to Hidden Consequencescited by 0
  12. 2026DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modelingcited by 0
1 more
  1. 2026Unfold The World: Factorize 4D Properties in Reinforcing Spatial Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2025Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment
  3. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  4. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  5. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2025RLHF in an SFT Way: From Optimal Solution to Reward-Weighted Alignment
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective