Data to 14 Oct 2026

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

arXiv 2502.10391 ↗First posted 14 Feb 2025cs.CL cs.CVcited by 83

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more

LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Despite notable advancements in Multimodal Large Language Models (MLLMs), most state-of-the-art models have not undergone thorough alignment with human preferences. This gap exists because current alignment research has primarily achieved progress in specific areas (e.g., hallucination reduction), while the broader question of whether aligning models with human preferences can systematically enhance MLLM capability remains largely unexplored. To this end, we introduce MM-RLHF, a dataset containing 120k fine-grained, human-annotated preference comparison pairs. This dataset represents a substantial advancement over existing resources, offering superior size, diversity, annotation granularity, and quality. Leveraging this dataset, we propose several key innovations to improve both the quality of reward models and the efficiency of alignment algorithms. Notably, we introduce a Critique-Based Reward Model, which generates critiques of model outputs before assigning scores, offering enhanced interpretability and more informative feedback compared to traditional scalar reward mechanisms. Additionally, we propose Dynamic Reward Scaling, a method that adjusts the loss weight of each sample according to the reward signal, thereby optimizing the use of high-quality comparison pairs. Our approach is rigorously evaluated across 10 distinct dimensions and 27 benchmarks, with results demonstrating significant and consistent improvements in model performance. Specifically, fine-tuning LLaVA-ov-7B with MM-RLHF and our alignment algorithm leads to a 19.5% increase in conversational abilities and a 60% improvement in safety. We have open-sourced the preference dataset, reward model, training and evaluation code, as well as reward modeling and safety benchmarks. For more details, please visit our project page: https://mm-rlhf.github.io.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 84 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

57 of 84 references.

Closest ancestors · 25

Same direction, newest first

13 more

Other references · 32

2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
2023Visual Instruction Tuningcited by 7,869
2023Mistral 7Bcited by 3,164
2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733

20 more

2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 135
2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
2024Direct Preference Optimization with an Offsetcited by 122
2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 100
2024β-DPO: Direct Preference Optimization with Dynamic βcited by 90
2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 86
2024Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Modelscited by 76
2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 71
2024Red Teaming Visual Language Modelscited by 65
2024VLSBench: Unveiling Visual Leakage in Multimodal Safetycited by 63
2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 40; 2026: 43.

83 papers.

Sort:

By area:

2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
2025Thyme: Think Beyond Imagescited by 167
2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
2025Kwai Keye-VL 1.5 Technical Reportcited by 73
2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
2025Kwai Keye-VL Technical Reportcited by 47
2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 30
2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
2025MME-VideoOCR: Evaluating OCR-Based Capabilities of Multimodal LLMs in Video Scenarioscited by 18

38 more

2025GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinkingcited by 15
2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 15
2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
2025Q-Ponder: A Unified Training Pipeline for Reasoning-based Visual Quality Assessmentcited by 14
2025Uni-Layout: Integrating Human Feedback in Unified Layout Generation and Evaluationcited by 14
2025Aligning Multimodal LLM with Human Preference: A Surveycited by 13
2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
2025VisualSphinx: Large-Scale Synthetic Vision Logic Puzzles for RLcited by 12
2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editingcited by 12
2025MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questionscited by 11
2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
2026PhyCritic: Multimodal Critic Models for Physical AIcited by 10
2025LAD-Reasoner: Tiny Multimodal Models are Good Reasoners for Logical Anomaly Detectioncited by 9
2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMscited by 9
2025Multi-speaker Attention Alignment for Multimodal Social Interactioncited by 9
2026CreatiParser: Generative Image Parsing of Raster Graphic Designs into Editable Layerscited by 9
2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
2025TIM-PRM: Verifying multimodal reasoning with Tool-Integrated PRMcited by 8
2025Self-alignment of Large Video Language Models with Refined Regularized Preference Optimizationcited by 7
2026Unified Generation and Self-Verification for Vision-Language Models via Advantage Decoupled Preference Optimizationcited by 7
2026LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoningcited by 7
2026Toward Native Multimodal Modeling: A Roadmapcited by 7
2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 6
2026MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimizationcited by 6
2025NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimationcited by 5
2026Visual Preference Optimization with Rubric Rewardscited by 5
2025Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spotscited by 4
2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 4
2025RADAR: A Risk-Aware Dynamic Multi-Agent Framework for LLM Safety Evaluation via Role-Specialized Collaborationcited by 4
2026MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Modelscited by 4
2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Modelscited by 4
2026Artifact-Bench: Evaluating MLLMs on Detecting and Assessing the Artifacts of AI-Generated Videoscited by 4
2026ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Searchcited by 3
2026BLM-Guard: Explainable Multimodal Ad Moderation with Chain-of-Thought and Policy-Aligned Rewardscited by 3
2026Decoding by Perturbation: Mitigating MLLM Hallucinations via Dynamic Textual Perturbationcited by 3

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way