mlsanityData to 7 Oct 2026

Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image

arXiv 2512.16899 ↗First posted 18 Dec 2025cs.CL cs.CVcited by 16

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Reward models (RMs) are essential for training large language models (LLMs), but remain underexplored for omni models that handle interleaved image and text sequences. We introduce Multimodal RewardBench 2 (MMRB2), the first comprehensive benchmark for reward models on multimodal understanding and (interleaved) generation. MMRB2 spans four tasks: text-to-image, image editing, interleaved generation, and multimodal reasoning ("thinking-with-images"), providing 1,000 expert-annotated preference pairs per task from 23 models and agents across 21 source tasks. MMRB2 is designed with: (1) practical but challenging prompts; (2) responses from state-of-the-art models and agents; and (3) preference pairs with strong human-expert consensus, curated via an ensemble filtering strategy. Using MMRB2, we study existing judges for each subtask, including multimodal LLM-as-a-judge and models trained with human preferences. The latest Gemini 3 Pro attains 75-80% accuracy. GPT-5 and Gemini 2.5 Pro reach 66-75% accuracy, compared to >90% for humans, yet surpass the widely used GPT-4o (59%). The best performing open-source model Qwen3-VL-32B achieves similar accuracies as Gemini 2.5 Flash (64%). We also show that MMRB2 performance strongly correlates with downstream task success using Best-of-N sampling and conduct an in-depth analysis that shows key areas to improve the reward models going forward.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 66 of its 94 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

66 of 94 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing30 Sep
  2. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling28 Sep
  3. 2025HPSv3: Towards Wide-Spectrum Human Preference Score5 Aug
  4. 2025Unified Reward Model for Multimodal Understanding and Generation7 Mar
  5. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models20 Feb
  6. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models26 Nov
  7. 2023Rich Human Feedback for Text-to-Image Generation15 Dec
  8. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2025Gemma 3 Technical Reportcited by 1,686
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
38 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  2. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  4. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  5. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  6. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  7. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  8. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  9. 2024Self-Rewarding Language Modelscited by 654
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  11. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  12. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 468
  13. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  14. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  15. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  16. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  17. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  18. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  19. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  20. 2023Emu Edit: Precise Image Editing via Recognition and Generation Taskscited by 321
  21. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  22. 2024Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Modelscited by 260
  23. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  24. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 191
  25. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  26. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  27. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  28. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  29. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 132
  30. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  31. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  32. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 130
  33. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Modelscited by 102
  34. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  35. 2025MindCube: Spatial Mental Modeling from Limited Viewscited by 91
  36. 2025OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generationcited by 81
  37. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  38. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026Unified Personalized Reward Model for Vision Generationcited by 11
  2. 2026RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Timecited by 11
  3. 2026Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteriacited by 8
  4. 2026Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modelingcited by 6
  5. 2026SpatialReward: Bridging the Perception Gap in Online RL for Image Editing via Explicit Spatial Reasoningcited by 5
  6. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Modelscited by 3
  7. 2026OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibrationcited by 2
  8. 2026Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Modelscited by 1
  9. 2026AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignmentcited by 1
  10. 2026SpatialFlow-GRPO: Where Spatial Credit Drives Image Editingcited by 1
  11. 2026Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklistcited by 0
  12. 2026ReCAST: Reward Credit Assignment across Timesteps for Online Diffusion Reinforcementcited by 0
4 more
  1. 2026Think Before You Score: Thinking Reward Model for Visual Generationcited by 0
  2. 2026CollageAttack: Exploiting Cross-Modal Alignment Flaws in T2I Models through Spatial Text Compositioncited by 0
  3. 2026A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversightcited by 0
  4. 2026VIEScore2: Unified Image Evaluation with Spatially Grounded Explanationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025BaseReward: A Strong Baseline for Multimodal Reward Model
  2. 2025VideoRewardBench: Comprehensive Evaluation of Multimodal Reward Models for Video Understanding
  3. 2026DeltaRubric: Generative Multimodal Reward Modeling via Joint Planning and Verification
  4. 2026Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis
  5. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model
  6. 2026DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modeling
  7. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoning
  8. 2025From Captions to Rewards (CAREVL): Leveraging Large Language Model Experts for Enhanced Reward Modeling in Large Vision-Language Models