mlsanityData to 14 Oct 2026

RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

arXiv 2408.12109 ↗First posted 22 Aug 2024cs.CV cs.CLcited by 15

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Large vision-language models (LVLMs) often fail to align with human preferences, leading to issues like generating misleading content without proper visual context (also known as hallucination). A promising solution to this problem is using human-preference alignment techniques, such as best-of-n sampling and reinforcement learning. However, these techniques face the difficulty arising from the scarcity of visual preference data, which is required to train a visual reward model (VRM). In this work, we continue the line of research. We present a Robust Visual Reward Model (RoVRM) which improves human-preference alignment for LVLMs. RoVRM leverages auxiliary textual preference data through a three-phase progressive training and optimal transport-based preference data selection to effectively mitigate the scarcity of visual preference data. We experiment with RoVRM on the commonly used vision-language tasks based on the LLaVA-1.5-7B and -13B models. Experimental results demonstrate that RoVRM consistently outperforms traditional VRMs. Furthermore, our three-phase progressive training and preference data selection approaches can yield consistent performance gains over ranking-based alignment techniques, such as direct preference optimization.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 53 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

38 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  2. 2023VILA: On Pre-training for Visual Language Models12 Dec
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  4. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  5. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data20 Aug

Other references · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  11. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  12. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
21 more
  1. 2018Object Hallucination in Image Captioningcited by 669
  2. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  3. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  4. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  5. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  6. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  7. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  8. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  11. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  12. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimizationcited by 225
  13. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  14. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  15. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  16. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 105
  17. 2023Jointly Training Large Autoregressive Multimodal Modelscited by 32
  18. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  19. 2021SPOT: A framework for selection of prototypes using optimal transportcited by 13
  20. 2023Introduction to Transformers: an NLP Perspectivecited by 10
  21. 2017A survey of dimensionality reduction techniques based on random projectioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 3.

15 papers.

Sort:

By area:

  1. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  2. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  3. 2025GRAM: A Generative Foundation Reward Model for Reward Generalizationcited by 25
  4. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  5. 2025Quality-Driven Curation of Remote Sensing Vision-Language Data via Learned Scoring Modelscited by 11
  6. 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Trainingcited by 9
  7. 2025SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgementcited by 7
  8. 2024LRHP: Learning Representations for Human Preferences via Preference Pairscited by 5
  9. 2024YingSound: Video-Guided Sound Effects Generation with Multi-modal Chain-of-Thought Controlscited by 5
  10. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  11. 2025HEAL: A Hypothesis-Based Preference-Aware Analysis Frameworkcited by 4
  12. 2025DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimizationcited by 2
3 more
  1. 2025The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward Modelscited by 2
  2. 2026SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Modelscited by 2
  3. 2026UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignmentcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
  2. 2024Calibrated Self-Rewarding Vision Language Models
  3. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
  4. 2026Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling
  5. 2024Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment
  6. 2025Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization
  7. 2025Transferring Textual Preferences to Vision-Language Understanding through Model Merging
  8. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation