# Think Before You Score: Thinking Reward Model for Visual Generation

arXiv: https://arxiv.org/abs/2609.37372 · HTML: https://mlsanity.com/paper/2609.37372/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 49 of its 59 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 49)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment](https://mlsanity.com/paper/2403.05135/) (2024)
- [Step1X-Edit: A Practical Framework for General Image Editing](https://mlsanity.com/paper/2504.17761/) (2025)
- [Diffusion Policy Policy Optimization](https://mlsanity.com/paper/2409.00588/) (2024)
- [ImgEdit: A Unified Image Editing Dataset and Benchmark](https://mlsanity.com/paper/2505.20275/) (2025)
- [HPSv3: Towards Wide-Spectrum Human Preference Score](https://mlsanity.com/paper/2508.03789/) (2025)
- [Unified Reward Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.05236/) (2025)
- [GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation](https://mlsanity.com/paper/2406.13743/) (2024)
- [Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing](https://mlsanity.com/paper/2504.02826/) (2025)
- [Monet: Reasoning in Latent Visual Space Beyond Images and Language](https://mlsanity.com/paper/2511.21395/) (2025)
- [KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models](https://mlsanity.com/paper/2505.16707/) (2025)
- [GPT-IMAGE-EDIT-1.5M: A Million-Scale, GPT-Generated Image Dataset](https://mlsanity.com/paper/2507.21033/) (2025)
- [TIIF-Bench: How Does Your T2I Model Follow Your Instructions?](https://mlsanity.com/paper/2506.02161/) (2025)
- [EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling](https://mlsanity.com/paper/2509.23909/) (2025)
- [EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing](https://mlsanity.com/paper/2509.26346/) (2025)
- [Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching](https://mlsanity.com/paper/2509.05952/) (2025)
- [PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Framework](https://mlsanity.com/paper/2506.10741/) (2025)
- [EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation](https://mlsanity.com/paper/2412.18150/) (2024)
- [Debiasing Multimodal Large Language Models via Penalization of Language Priors](https://mlsanity.com/paper/2403.05262/) (2024)
- [Easier Painting Than Thinking: Can Text-to-Image Models Set the Stage, but Not Direct the Play?](https://mlsanity.com/paper/2509.03516/) (2025)
- [RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark](https://mlsanity.com/paper/2509.24897/) (2025)
- [Mavors: Multi-granularity Video Representation for Multimodal Large Language Model](https://mlsanity.com/paper/2504.10068/) (2025)
- [OpenGPT-4o-Image: A Comprehensive Dataset for Advanced Image Generation and Editing](https://mlsanity.com/paper/2509.24900/) (2025)

## Abstract (arXiv)

Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and candidate outputs directly to scalar rewards, leaving implicit what should be evaluated for each individual case. We introduce Think Before You Score, a paradigm that explicitly determines what matters for each case before judging how well the candidate performs. Following this principle, we propose the Thinking Reward Model (TRM), which formulates case-adaptive rubrics, performs rubric-guided assessment, and produces fine-grained pointwise rewards. We further observe that conventional pairwise preference optimization can induce score polarization, and introduce Pairwise Dual-Group Relative Policy Optimization (PD-GRPO), which leverages pairwise supervision to improve reward discrimination while preserving fine-grained pointwise scoring. Extensive experiments on image generation and editing reward-modeling benchmarks demonstrate that TRM achieves state-of-the-art performance among open-source reward models while remaining highly competitive with proprietary alternatives. Moreover, using TRM as a reward for reinforcement learning consistently improves diverse visual generation models, demonstrating that its fine-grained, case-adaptive rewards translate into effective optimization signals for visual generation.