# Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modeling

arXiv: https://arxiv.org/abs/2605.05922 · HTML: https://mlsanity.com/paper/2605.05922/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 21 of 21)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [Improving Video Generation with Human Feedback](https://mlsanity.com/paper/2501.13918/) (2025)
- [Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps](https://mlsanity.com/paper/2501.09732/) (2025)
- [Unified Reward Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.05236/) (2025)
- [Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.03318/) (2025)
- [LongCat-Video Technical Report](https://mlsanity.com/paper/2510.22200/) (2025)
- [RewardDance: Reward Scaling in Visual Generation](https://mlsanity.com/paper/2509.08826/) (2025)
- [LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment](https://mlsanity.com/paper/2412.04814/) (2024)
- [Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search](https://mlsanity.com/paper/2501.19252/) (2025)
- [VideoScore2: Think before You Score in Generative Video Evaluation](https://mlsanity.com/paper/2509.22799/) (2025)
- [The Dawn of Video Generation: Preliminary Explorations with SORA-like Models](https://mlsanity.com/paper/2410.05227/) (2024)
- [VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoning](https://mlsanity.com/paper/2510.10518/) (2025)
- [Rectifying Shortcut Behaviors in Preference-based Reward Learning](https://mlsanity.com/paper/2510.19050/) (2025)
- [VL Norm: Rethink Loss Aggregation in RLVR](https://mlsanity.com/paper/2509.07558/) (2025)

## Built on by (most cited first, 2 of 2)

- [RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling](https://mlsanity.com/paper/2609.22947/) (2026)
- [AgenticGen: Reward-Guided Agentic Video Generation for Advertising](https://mlsanity.com/paper/2609.09187/) (2026)

## Abstract (arXiv)

Recent advances in generative video models are increasingly driven by post-training and test-time scaling, both of which critically depend on the quality of video reward models (RMs). An ideal reward model should predict accurate rewards that align with human preferences across diverse scenarios. However, existing paradigms face a fundamental dilemma: textitDiscriminative RMs regress rewards directly on features extracted by multimodal large language models (MLLMs) without explicit reasoning, making them prone to shortcut learning and heavily reliant on massive data scaling for generalization. In contrast, textitGenerative RMs with Chain-of-Thought (CoT) reasoning exhibit superior interpretability and generalization potential, as they leverage fine-grained semantic supervision to internalize the rationales behind human preferences. However, they suffer from inherent optimization bottlenecks due to the coupling of reasoning and scoring within a single autoregressive inference chain. To harness the generalization benefits of CoT reasoning while mitigating the training instability of coupled reasoning and scoring, we introduce DeScore, a training-efficient and generalizable video reward model. DeScore employs a decoupled ``think-then-score'' paradigm: an MLLM first generates an explicit CoT, followed by a dedicated discriminative scoring module consisting of a learnable query token and a regression head that predicts the final reward. DeScore is optimized via a two-stage framework: (1) a discriminative cold start incorporating a random mask mechanism to ensure robust scoring capabilities, and (2) a dual-objective reinforcement learning stage that independently refines CoT reasoning quality and calibrates the final reward, ensuring that higher-quality reasoning directly translates to superior model performance.