# Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image

arXiv: https://arxiv.org/abs/2512.16899 · HTML: https://mlsanity.com/paper/2512.16899/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 66 of its 94 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 66)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [RewardBench: Evaluating Reward Models for Language Modeling](https://mlsanity.com/paper/2403.13787/) (2024)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering](https://mlsanity.com/paper/2303.11897/) (2023)
- [Fine-Grained Human Feedback Gives Better Rewards for Language Model Training](https://mlsanity.com/paper/2306.01693/) (2023)
- [BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset](https://mlsanity.com/paper/2505.09568/) (2025)

## Built on by (most cited first, 20 of 20)

- [RationalRewards: Reasoning Rewards Scale Visual Generation Both Training and Test Time](https://mlsanity.com/paper/2604.11626/) (2026)
- [Unified Personalized Reward Model for Vision Generation](https://mlsanity.com/paper/2602.02380/) (2026)
- [Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria](https://mlsanity.com/paper/2605.08354/) (2026)
- [SpatialReward: Bridging the Perception Gap in Online RL for Image Editing via Explicit Spatial Reasoning](https://mlsanity.com/paper/2602.07458/) (2026)
- [Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling](https://mlsanity.com/paper/2605.13062/) (2026)
- [Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Models](https://mlsanity.com/paper/2602.07533/) (2026)
- [AutoRubric-T2I: Robust Rule-Based Reward Model for Text-to-Image Alignment](https://mlsanity.com/paper/2605.17602/) (2026)
- [OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibration](https://mlsanity.com/paper/2605.28805/) (2026)
- [SpatialFlow-GRPO: Where Spatial Credit Drives Image Editing](https://mlsanity.com/paper/2606.26872/) (2026)
- [Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist](https://mlsanity.com/paper/2606.31711/) (2026)
- [Seeing Isn't Believing: Uncovering Blind Spots in Evaluator Vision-Language Models](https://mlsanity.com/paper/2604.21523/) (2026)
- [ReCAST: Reward Credit Assignment across Timesteps for Online Diffusion Reinforcement](https://mlsanity.com/paper/2609.13425/) (2026)
- [Think Before You Score: Thinking Reward Model for Visual Generation](https://mlsanity.com/paper/2609.37372/) (2026)
- [CollageAttack: Exploiting Cross-Modal Alignment Flaws in T2I Models through Spatial Text Composition](https://mlsanity.com/paper/2609.38253/) (2026)
- [A Low Grounding Score Is Not an Ungrounded Judge: Identifying the Perceptibility Confound in Multimodal Oversight](https://mlsanity.com/paper/2610.00111/) (2026)
- [VIEScore2: Unified Image Evaluation with Spatially Grounded Explanations](https://mlsanity.com/paper/2610.00994/) (2026)
- [Post-Training Frontier Text-to-Image Models by Composing Preference and Rubric Rewards](https://mlsanity.com/paper/2610.02967/) (2026)
- [Unsupervised Multimodal Reward Modeling](https://mlsanity.com/paper/2610.12707/) (2026)
- [DuplexBridge: Coordinating Understanding and Generation for Unified Image Editing](https://mlsanity.com/paper/2610.18819/) (2026)
- [How Far Can Coding Agents Go in Visual Generation?](https://mlsanity.com/paper/2610.19025/) (2026)

## Abstract (arXiv)

Reward models (RMs) are essential for training large language models (LLMs), but remain underexplored for omni models that handle interleaved image and text sequences. We introduce Multimodal RewardBench 2 (MMRB2), the first comprehensive benchmark for reward models on multimodal understanding and (interleaved) generation. MMRB2 spans four tasks: text-to-image, image editing, interleaved generation, and multimodal reasoning ("thinking-with-images"), providing 1,000 expert-annotated preference pairs per task from 23 models and agents across 21 source tasks. MMRB2 is designed with: (1) practical but challenging prompts; (2) responses from state-of-the-art models and agents; and (3) preference pairs with strong human-expert consensus, curated via an ensemble filtering strategy. Using MMRB2, we study existing judges for each subtask, including multimodal LLM-as-a-judge and models trained with human preferences. The latest Gemini 3 Pro attains 75-80% accuracy. GPT-5 and Gemini 2.5 Pro reach 66-75% accuracy, compared to >90% for humans, yet surpass the widely used GPT-4o (59%). The best performing open-source model Qwen3-VL-32B achieves similar accuracies as Gemini 2.5 Flash (64%). We also show that MMRB2 performance strongly correlates with downstream task success using Best-of-N sampling and conduct an in-depth analysis that shows key areas to improve the reward models going forward.