mlsanityData to 7 Oct 2026

Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models

arXiv 2502.14191 ↗First posted 20 Feb 2025cs.CV cs.AIcited by 40

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Reward models play an essential role in training vision-language models (VLMs) by assessing output quality to enable aligning with human preferences. Despite their importance, the research community lacks comprehensive open benchmarks for evaluating multimodal reward models in VLMs. To address this gap, we introduce Multimodal RewardBench, an expert-annotated benchmark covering six domains: general correctness, preference, knowledge, reasoning, safety, and visual question-answering. Our dataset comprises 5,211 annotated (prompt, chosen response, rejected response) triplets collected from various VLMs. In evaluating a range of VLM judges, we find that even the top-performing models, Gemini 1.5 Pro and Claude 3.5 Sonnet, achieve only 72% overall accuracy. Notably, most models struggle in the reasoning and safety domains. These findings suggest that Multimodal RewardBench offers a challenging testbed for advancing reward model development across multiple domains. We release the benchmark at https://github.com/facebookresearch/multimodal_rewardbench.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 44 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

26 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark9 Jan
  2. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  3. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  4. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  9. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  10. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  11. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  12. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
8 more
  1. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  2. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 136
  3. 2024Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluationcited by 123
  4. 2022Holistic Evaluation of Language Modelscited by 115
  5. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  6. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  7. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Modelscited by 18
  8. 2023Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 22; 2026: 18.

40 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  3. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  4. 2025Reinforcement Learning from Human Feedbackcited by 53
  5. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 20
  6. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  7. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  8. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agentcited by 16
  9. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 16
  10. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  11. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  12. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
28 more
  1. 2025PPTArena: A Benchmark for PowerPoint Editingcited by 9
  2. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 9
  3. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  4. 2025Self-Improving VLM Judges Without Human Annotationscited by 7
  5. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  6. 2025Activation Reward Models for Few-Shot Model Alignmentcited by 6
  7. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5
  8. 2025ReasonX: MLLM-Guided Intrinsic Image Decompositioncited by 5
  9. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  10. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  11. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  12. 2025Libra: Assessing and Improving Reward Model by Learning to Thinkcited by 4
  13. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  14. 2026Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generationcited by 3
  15. 2026Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Modelscited by 3
  16. 2026Building a Precise Video Language with Human-AI Oversightcited by 3
  17. 2026Meta-CoT: Enhancing Granularity and Generalization in Image Editingcited by 3
  18. 2026MJ1: Multimodal Judgment via Grounded Verificationcited by 2
  19. 2026Long-form RewardBench: Evaluating Reward Models for Long-form Generationcited by 2
  20. 2025Un-Doubling Diffusion: LLM-guided Disambiguation of Homonym Duplicationcited by 1
  21. 2026Entropy-Guided Data-Efficient Training for Multimodal Reasoning Reward Modelscited by 1
  22. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Modelscited by 1
  23. 2025When Slower Isn't Truer: Inverse Scaling Law of Truthfulness in Multimodal Reasoningcited by 0
  24. 2026DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modelingcited by 0
  25. 2026QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agentscited by 0
  26. 2026HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentationcited by 0
  27. 2026Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMscited by 0
  28. 2026A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversightcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
  2. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models
  3. 2025Transferring Textual Preferences to Vision-Language Understanding through Model Merging
  4. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning
  5. 2025VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
  6. 2026Lost in Translation: Do LVLM Judges Generalize Across Languages?
  7. 2025Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned
  8. 2025Revisiting the Learning Objectives of Vision-Language Reward Models