mlsanityData to 7 Oct 2026

VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models

arXiv 2411.17451 ↗First posted 26 Nov 2024cs.CV cs.CLcited by 85

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Vision-language generative reward models (VL-GenRMs) play a crucial role in aligning and evaluating multimodal AI systems, yet their own evaluation remains under-explored. Current assessment methods primarily rely on AI-annotated preference labels from traditional VL tasks, which can introduce biases and often fail to effectively challenge state-of-the-art models. To address these limitations, we introduce VL-RewardBench, a comprehensive benchmark spanning general multimodal queries, visual hallucination detection, and complex reasoning tasks. Through our AI-assisted annotation pipeline that combines sample selection with human verification, we curate 1,250 high-quality examples specifically designed to probe VL-GenRMs limitations. Comprehensive evaluation across 16 leading large vision-language models demonstrates VL-RewardBench's effectiveness as a challenging testbed, where even GPT-4o achieves only 65.4% accuracy, and state-of-the-art open-source models such as Qwen2-VL-72B, struggle to surpass random-guessing. Importantly, performance on VL-RewardBench strongly correlates (Pearson's r > 0.9) with MMMU-Pro accuracy using Best-of-N sampling with VL-GenRMs. Analysis experiments uncover three critical insights for improving VL-GenRMs: (i) models predominantly fail at basic visual perception tasks rather than reasoning tasks; (ii) inference-time scaling benefits vary dramatically by model capacity; and (iii) training VL-GenRMs to learn to judge substantially boosts judgment capability (+14.7% accuracy for a 7B VL-GenRM). We believe VL-RewardBench along with the experimental insights will become a valuable resource for advancing VL-GenRMs.

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 73 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

62 of 73 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment12 Oct
  2. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?5 Jul
  3. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback7 Apr
  4. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuning18 Feb
  5. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  6. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
38 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  5. 2024Qwen2 Technical Reportcited by 2,076
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  12. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  13. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  14. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  15. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  16. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  17. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  18. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  19. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  20. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  21. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  22. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  23. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  24. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 336
  25. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  26. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  27. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  28. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  29. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  30. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  31. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  32. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 118
  33. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  34. 2024Calibrated Self-Rewarding Vision Language Modelscited by 77
  35. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  36. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  37. 2023Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Modelscited by 66
  38. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 53; 2026: 31.

85 papers.

Sort:

By area:

  1. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 233
  2. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  3. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  4. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  5. 2025MiMo-VL Technical Reportcited by 106
  6. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 86
  7. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignmentcited by 79
  8. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  9. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2025Reinforcement Learning from Human Feedbackcited by 53
  12. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
38 more
  1. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
  2. 2025LLaVA-Critic-R1: Your Critic Model is Secretly a Strong Policy Modelcited by 34
  3. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  4. 2025Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)cited by 23
  5. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  6. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  7. 2025ViLBench: A Suite for Vision-Language Process Reward Modelingcited by 16
  8. 2025Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Imagecited by 16
  9. 2025BaseReward: A Strong Baseline for Multimodal Reward Modelcited by 15
  10. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  11. 2025CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metricscited by 14
  12. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 14
  13. 2025Aligning Multimodal LLM with Human Preference: A Surveycited by 12
  14. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Modelcited by 12
  15. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  16. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Modelscited by 10
  17. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Programcited by 10
  18. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Taskscited by 9
  19. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 9
  20. 2025Position: Reasoning After Perception Means Reasoning Without Visioncited by 8
  21. 2025Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesiscited by 8
  22. 2025MR. Judge: Multimodal Reasoner as a Judgecited by 7
  23. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
  24. 2025Self-Improving VLM Judges Without Human Annotationscited by 7
  25. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluationcited by 7
  26. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesiscited by 7
  27. 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluationcited by 7
  28. 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluatorscited by 6
  29. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understandingcited by 5
  30. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  31. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  32. 2026Visual Preference Optimization with Rubric Rewardscited by 5
  33. 2025Efficient MAP Estimation of LLM Judgment Performance with Prior Transfercited by 4
  34. 2025VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videoscited by 4
  35. 2025VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understandingcited by 4
  36. 2025RoleRMBench & RoleRM: Towards Reward Modeling for Profile-Based Role Play in Dialogue Systemscited by 4
  37. 2025Transferring Textual Preferences to Vision-Language Understanding through Model Mergingcited by 3
  38. 2025Sightation Counts: Leveraging Sighted User Feedback in Building a BLV-aligned Dataset of Diagram Descriptionscited by 3

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling
  2. 2025Revisiting the Learning Objectives of Vision-Language Reward Models
  3. 2025Training Vision-Language Process Reward Models for Test-Time Scaling in Multimodal Reasoning: Key Insights and Lessons Learned
  4. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
  5. 2026Lost in Translation: Do LVLM Judges Generalize Across Languages?
  6. 2025ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
  7. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
  8. 2025Self-Rewarding Vision-Language Model via Reasoning Decomposition