# Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models

arXiv: https://arxiv.org/abs/2502.14191 · HTML: https://mlsanity.com/paper/2502.14191/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [Aria: An Open Multimodal Native Mixture-of-Experts Model](https://mlsanity.com/paper/2410.05993/) (2024)
- [LLaVA-Critic: Learning to Evaluate Multimodal Models](https://mlsanity.com/paper/2410.02712/) (2024)
- [Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark](https://mlsanity.com/paper/2501.05444/) (2025)
- [Holistic Evaluation of Language Models](https://mlsanity.com/paper/2211.09110/) (2022)
- [Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation](https://mlsanity.com/paper/2401.06591/) (2024)
- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models](https://mlsanity.com/paper/2411.17451/) (2024)
- [VHELM: A Holistic Evaluation of Vision Language Models](https://mlsanity.com/paper/2410.07112/) (2024)
- [Image2Struct: Benchmarking Structure Extraction for Vision-Language Models](https://mlsanity.com/paper/2410.22456/) (2024)
- [Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models](https://mlsanity.com/paper/2305.17311/) (2023)

## Built on by (most cited first, 30 of 44)

- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://mlsanity.com/paper/2503.09567/) (2025)
- [RewardBench 2: Advancing Reward Model Evaluation](https://mlsanity.com/paper/2506.01937/) (2025)
- [R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning](https://mlsanity.com/paper/2505.02835/) (2025)
- [Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2504.12501/) (2025)
- [Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark](https://mlsanity.com/paper/2506.04280/) (2025)
- [Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image](https://mlsanity.com/paper/2512.16899/) (2025)
- [VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models](https://mlsanity.com/paper/2503.07478/) (2025)
- [CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agent](https://mlsanity.com/paper/2510.18596/) (2025)
- [ViLBench: A Suite for Vision-Language Process Reward Modeling](https://mlsanity.com/paper/2503.20271/) (2025)
- [BaseReward: A Strong Baseline for Multimodal Reward Model](https://mlsanity.com/paper/2509.16127/) (2025)
- [Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following](https://mlsanity.com/paper/2511.21662/) (2025)
- [Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences](https://mlsanity.com/paper/2510.23451/) (2025)
- [PPTArena: A Benchmark for PowerPoint Editing](https://mlsanity.com/paper/2512.03042/) (2025)
- [PhyCritic: Multimodal Critic Models for Physical AI](https://mlsanity.com/paper/2602.11124/) (2026)
- [Enhancing Large Language Model Reasoning with Reward Models: An Analytical Survey](https://mlsanity.com/paper/2510.01925/) (2025)
- [Self-Improving VLM Judges Without Human Annotations](https://mlsanity.com/paper/2512.05145/) (2025)
- [Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis](https://mlsanity.com/paper/2602.00846/) (2026)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [Activation Reward Models for Few-Shot Model Alignment](https://mlsanity.com/paper/2507.01368/) (2025)
- [VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understanding](https://mlsanity.com/paper/2509.00484/) (2025)
- [ReasonX: MLLM-Guided Intrinsic Image Decomposition](https://mlsanity.com/paper/2512.04222/) (2025)
- [MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learning](https://mlsanity.com/paper/2603.25108/) (2026)
- [Visual Preference Optimization with Rubric Rewards](https://mlsanity.com/paper/2604.13029/) (2026)
- [Libra: Assessing and Improving Reward Model by Learning to Think](https://mlsanity.com/paper/2507.21645/) (2025)
- [RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systems](https://mlsanity.com/paper/2512.10575/) (2025)
- [Rationale Matters: Learning Transferable Rubrics via Proxy-Guided Critique for VLM Reward Models](https://mlsanity.com/paper/2603.16600/) (2026)
- [Advancing Multimodal Judge Models through a Capability-Oriented Benchmark and MCTS-Driven Data Generation](https://mlsanity.com/paper/2603.00546/) (2026)
- [MJ1: Multimodal Judgment via Grounded Verification](https://mlsanity.com/paper/2603.07990/) (2026)
- [Building a Precise Video Language with Human-AI Oversight](https://mlsanity.com/paper/2604.21718/) (2026)
- [Meta-CoT: Enhancing Granularity and Generalization in Image Editing](https://mlsanity.com/paper/2604.24625/) (2026)

## Abstract (arXiv)

Reward models play an essential role in training vision-language models (VLMs) by assessing output quality to enable aligning with human preferences. Despite their importance, the research community lacks comprehensive open benchmarks for evaluating multimodal reward models in VLMs. To address this gap, we introduce Multimodal RewardBench, an expert-annotated benchmark covering six domains: general correctness, preference, knowledge, reasoning, safety, and visual question-answering. Our dataset comprises 5,211 annotated (prompt, chosen response, rejected response) triplets collected from various VLMs. In evaluating a range of VLM judges, we find that even the top-performing models, Gemini 1.5 Pro and Claude 3.5 Sonnet, achieve only 72% overall accuracy. Notably, most models struggle in the reasoning and safety domains. These findings suggest that Multimodal RewardBench offers a challenging testbed for advancing reward model development across multiple domains. We release the benchmark at https://github.com/facebookresearch/multimodal_rewardbench.