VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understanding

arXiv 2509.00484 ↗First posted 30 Aug 2025cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

also in 1 more

LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Multimodal reward models (MRMs) play a crucial role in the training, inference, and evaluation of Large Vision Language Models (LVLMs) by assessing response quality. However, existing benchmarks for evaluating MRMs in the video domain suffer from a limited number and diversity of questions, a lack of comprehensive evaluation dimensions, and inadequate evaluation of diverse types of MRMs. To address these gaps, we introduce VideoRewardBench, the first comprehensive benchmark covering four core aspects of video understanding: perception, knowledge, reasoning, and safety. Through our AI-assisted data pipeline, we curate a high-quality preference dataset of 1,563 annotated samples, including 1,482 unique videos and 1,559 distinct questions--15 times the number found in the most question-rich prior benchmark. Each sample is a triplet consisting of a video-text prompt, a chosen response, and a rejected response. We also conduct a comprehensive evaluation across 28 multimodal reward models spanning three categories: generative, discriminative, and semi-scalar. Results show that even the top-performing model GPT-4o achieves only 57.0% overall accuracy, and the state-of-the-art open-source model Qwen2.5-VL-72B reaches merely 53.3%. Our analysis further reveals three key insights: (i) MRMs trained with reinforcement learning (RL) do not necessarily exhibit stronger cross-modal generalization than those trained without RL; (ii) except for discriminative MRMs, other types of MRMs across varying model capacities can benefit from inference-time scaling; and (iii) variations in input video frame count have different effects on different types of MRMs. We believe VideoRewardBench offers a challenging and valuable benchmark for advancing the evaluation and development of MRMs in the video domain.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 45 references that are in the map. First posted on arXiv 30 Aug 2025.

Builds on

36 of 45 references.

Closest ancestors · 4

Same direction, newest first

Other references · 32

2022Training language models to follow instructions with human feedbackcited by 15,712
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
2025Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videoscited by 225
2025Unified Reward Model for Multimodal Understanding and Generationcited by 194

20 more

2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language modelscited by 35
2025Judge Anything: MLLM as a Judge Across Any Modalitycited by 33
2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 30
2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25
2025Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMscited by 25
2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Modelscited by 10
2025MR. Judge: Multimodal Reasoner as a Judgecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
2026Social Caption: Evaluating Social Understanding in Multimodal Modelscited by 2
2026You Only Judge Once: Multi-response Reward Modeling in a Single Forward Passcited by 0
2026Video Understanding Reward Modeling: A Robust Benchmark and Performant Reward Modelscited by 0
2026VlogReward: Learning Multi-Dimensional Evaluation for Vlog Editingcited by 0

Similar, not cited

Close in wording, no citation either way