# DenseGRPO: From Sparse to Dense Reward for Flow Matching Model Alignment arXiv: https://arxiv.org/abs/2601.20218 · HTML: https://mlsanity.com/paper/2601.20218/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. - Builds on 34 of its 38 references that are in the map. - First posted on arXiv 28 Jan 2026. ## Placed in - LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/) - Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/) - LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/) ## Builds on (most cited first, 30 of 34) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023) - [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022) - [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022) - [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022) - [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023) - [FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space](https://mlsanity.com/paper/2506.15742/) (2025) - [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023) - [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023) - [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023) - [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023) - [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025) - [Aligning Text-to-Image Models using Human Feedback](https://mlsanity.com/paper/2302.12192/) (2023) - [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025) - [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023) - [Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model](https://mlsanity.com/paper/2311.13231/) (2023) - [Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization](https://mlsanity.com/paper/2210.01241/) (2022) - [Unified Reward Model for Multimodal Understanding and Generation](https://mlsanity.com/paper/2503.05236/) (2025) - [DreamVideo: Composing Your Dream Videos with Customized Subject and Motion](https://mlsanity.com/paper/2312.04433/) (2023) - [MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDE](https://mlsanity.com/paper/2507.21802/) (2025) - [Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distribution](https://mlsanity.com/paper/2501.11561/) (2025) - [TempFlow-GRPO: When Timing Matters for GRPO in Flow Models](https://mlsanity.com/paper/2508.04324/) (2025) - [Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning](https://mlsanity.com/paper/2508.20751/) (2025) - [Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Model](https://mlsanity.com/paper/2503.07703/) (2025) - [A Survey of Temporal Credit Assignment in Deep Reinforcement Learning](https://mlsanity.com/paper/2312.01072/) (2023) - [A Dense Reward View on Aligning Text-to-Image Diffusion with Preference](https://mlsanity.com/paper/2402.08265/) (2024) - [Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedback](https://mlsanity.com/paper/2412.02617/) (2024) ## Built on by (most cited first, 21 of 21) - [Reinforce Adjoint Matching: Scaling RL Post-Training of Diffusion and Flow-Matching Models](https://mlsanity.com/paper/2605.10759/) (2026) - [Wan-Image: Pushing the Boundaries of Generative Visual Intelligence](https://mlsanity.com/paper/2604.19858/) (2026) - [Unified Personalized Reward Model for Vision Generation](https://mlsanity.com/paper/2602.02380/) (2026) - [UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation](https://mlsanity.com/paper/2603.23500/) (2026) - [Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO](https://mlsanity.com/paper/2602.06422/) (2026) - [Diffusion Reinforcement Learning via Centered Reward Distillation](https://mlsanity.com/paper/2603.14128/) (2026) - [FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling](https://mlsanity.com/paper/2604.06916/) (2026) - [Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping](https://mlsanity.com/paper/2605.10937/) (2026) - [Advances in GRPO for Generation Models: A Survey](https://mlsanity.com/paper/2603.06623/) (2026) - [AdvDMD: Adversarial Reward Meets DMD For High-Quality Few-Step Generation](https://mlsanity.com/paper/2604.28126/) (2026) - [Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoE](https://mlsanity.com/paper/2606.26938/) (2026) - [Think over Trajectories: Leveraging Video Generation to Reconstruct GPS Trajectories from Cellular Signaling](https://mlsanity.com/paper/2603.26610/) (2026) - [Explicit Critic Guidance for Aligning Diffusion Models](https://mlsanity.com/paper/2605.27736/) (2026) - [HPO: Hysteretic Policy Optimization for Stable and Efficient Training under Sparse-Reward Regime](https://mlsanity.com/paper/2605.30201/) (2026) - [Counterfactual Shapley Credit Assignment](https://mlsanity.com/paper/2607.16999/) (2026) - [LC-GRPO: Bridging Train-Inference Gap for Flow-Based GRPO with Langevin Correction](https://mlsanity.com/paper/2608.05600/) (2026) - [LeanGRPO: Eliminating Redundant Recomputation in Diffusion RL](https://mlsanity.com/paper/2609.03528/) (2026) - [λ-Controlled GRPO: Turning Flow-Matching Ratio Instability into a Budgeted Resource](https://mlsanity.com/paper/2609.22041/) (2026) - [ExploreNet: Learning Where to Explore in Diffusion GRPO](https://mlsanity.com/paper/2609.38329/) (2026) - [CAST: Causal Advantage-Structured Training with Spatially Grounded Compositional Rewards for Diffusion Models](https://mlsanity.com/paper/2609.39441/) (2026) - [Flow Matching Reinforcement for 3D Mesh Generation via Dynamic Homing Optimization](https://mlsanity.com/paper/2610.01233/) (2026) ## Abstract (arXiv) Recent GRPO-based approaches built on flow matching models have shown remarkable improvements in human preference alignment for text-to-image generation. Nevertheless, they still suffer from the sparse reward problem: the terminal reward of the entire denoising trajectory is applied to all intermediate steps, resulting in a mismatch between the global feedback signals and the exact fine-grained contributions at intermediate denoising steps. To address this issue, we introduce DenseGRPO, a novel framework that aligns human preference with dense rewards, which evaluates the fine-grained contribution of each denoising step. Specifically, our approach includes two key components: (1) we propose to predict the step-wise reward gain as dense reward of each denoising step, which applies a reward model on the intermediate clean images via an ODE-based approach. This manner ensures an alignment between feedback signals and the contributions of individual steps, facilitating effective training; and (2) based on the estimated dense rewards, a mismatch drawback between the uniform exploration setting and the time-varying noise intensity in existing GRPO-based methods is revealed, leading to an inappropriate exploration space. Thus, we propose a reward-aware scheme to calibrate the exploration space by adaptively adjusting a timestep-specific stochasticity injection in the SDE sampler, ensuring a suitable exploration space at all timesteps. Extensive experiments on multiple standard benchmarks demonstrate the effectiveness of the proposed DenseGRPO and highlight the critical role of the valid dense rewards in flow matching model alignment.