mlsanityData to 14 Oct 2026

RewardDance: Reward Scaling in Visual Generation

arXiv 2509.08826 ↗First posted 10 Sep 2025cs.CVcited by 68

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Reward Models (RMs) are critical for improving generation models via Reinforcement Learning (RL), yet the RM scaling paradigm in visual generation remains largely unexplored. It primarily due to fundamental limitations in existing approaches: CLIP-based RMs suffer from architectural and input modality constraints, while prevalent Bradley-Terry losses are fundamentally misaligned with the next-token prediction mechanism of Vision-Language Models (VLMs), hindering effective scaling. More critically, the RLHF optimization process is plagued by Reward Hacking issue, where models exploit flaws in the reward signal without improving true quality. To address these challenges, we introduce RewardDance, a scalable reward modeling framework that overcomes these barriers through a novel generative reward paradigm. By reformulating the reward score as the model's probability of predicting a "yes" token, indicating that the generated image outperforms a reference image according to specific criteria, RewardDance intrinsically aligns reward objectives with VLM architectures. This alignment unlocks scaling across two dimensions: (1) Model Scaling: Systematic scaling of RMs up to 26 billion parameters; (2) Context Scaling: Integration of task-specific instructions, reference examples, and chain-of-thought (CoT) reasoning. Extensive experiments demonstrate that RewardDance significantly surpasses state-of-the-art methods in text-to-image, text-to-video, and image-to-video generation. Crucially, we resolve the persistent challenge of "reward hacking": Our large-scale RMs exhibit and maintain high reward variance during RL fine-tuning, proving their resistance to hacking and ability to produce diverse, high-quality outputs. It greatly relieves the mode collapse problem that plagues smaller models.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 58 of its 70 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

58 of 70 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025HPSv3: Towards Wide-Spectrum Human Preference Score5 Aug
  2. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  3. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning6 May
  4. 2025Improving Video Generation with Human Feedback23 Jan
  5. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation30 Dec
  6. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimization19 Dec
  7. 2024ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback11 Apr
  8. 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
  9. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preference25 Mar

Other references · 49

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Flow Matching for Generative Modelingcited by 5,702
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  12. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
37 more
  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  4. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  5. 2025Qwen-Image Technical Reportcited by 1,044
  6. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  7. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  8. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  9. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  10. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  11. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  13. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  14. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 259
  15. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  16. 2025Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Stepscited by 234
  17. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 168
  18. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 160
  19. 2025Seedream 3.0 Technical Reportcited by 154
  20. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  21. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 109
  22. 2025Seaweed-7B: Cost-Effective Training of Video Generation Foundation Modelcited by 82
  23. 2023An Inverse Scaling Law for CLIP Trainingcited by 77
  24. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  25. 2025Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Modelcited by 62
  26. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  27. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  28. 2024Schedule On the Fly: Diffusion Time Prediction for Faster and Better Image Generationcited by 30
  29. 2025A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuningcited by 30
  30. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimizationcited by 21
  31. 2024Multi-Reward as Condition for Instruction-based Image Editingcited by 20
  32. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Modelscited by 19
  33. 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editingcited by 18
  34. 2025WorldPM: Scaling Human Preference Modelingcited by 13
  35. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  36. 2024ByteEdit: Boost, Comply and Accelerate Generative Image Editingcited by 10
  37. 2024UniFL: Improve Latent Diffusion Model via Unified Feedback Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 18; 2026: 50.

68 papers.

Sort:

By area:

  1. 2025Seedream 4.0: Toward Next-generation Multimodal Image Generationcited by 249
  2. 2026Seedance 2.0: Advancing Video Generation for World Complexitycited by 160
  3. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 86
  4. 2025Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Modelcited by 80
  5. 2025Uniworld-V2: Reinforce Image Editing with Diffusion Negative-aware Finetuning and MLLM Implicit Feedbackcited by 79
  6. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 64
  7. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 50
  8. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 49
  9. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 40
  10. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  11. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimizationcited by 16
  12. 2025Hyper-Bagel: A Unified Acceleration Framework for Multimodal Understanding and Generationcited by 13
38 more
  1. 2026Unified Personalized Reward Model for Vision Generationcited by 13
  2. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  3. 2025VR-Thinker: Boosting Video Reward Models through Thinking-with-Image Reasoningcited by 12
  4. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  5. 2026Understanding Reward Hacking in Text-to-Image Reinforcement Learningcited by 9
  6. 2026Leveraging Verifier-Based Reinforcement Learning in Image Editingcited by 9
  7. 2025MoGAN: Improving Motion Quality in Video Diffusion via Few-Step Motion Adversarial Post-Trainingcited by 8
  8. 2025Towards Reason-Informed Video Editing in Unified Models with Self-Reflective Learningcited by 7
  9. 2026Endogenous Reprompting: Self-Evolving Cognitive Alignment for Unified Multimodal Modelscited by 7
  10. 2026SpatialReward: Bridging the Perception Gap in Online RL for Image Editing via Explicit Spatial Reasoningcited by 7
  11. 2025Rethinking Reward Signals in Video GRPO: When Scores Become Targetscited by 6
  12. 2026Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modelingcited by 6
  13. 2025ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinementcited by 5
  14. 2026Fine-T2I: An Open, Large-Scale, and Diverse Dataset for High-Quality T2I Fine-Tuningcited by 5
  15. 2026A Systematic Post-Train Framework for Video Generationcited by 5
  16. 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
  17. 2026CaC: Advancing Video Reward Models via Hierarchical Spatiotemporal Concentratingcited by 5
  18. 2026Qwen-Image-2.0-RL Technical Reportcited by 5
  19. 2025SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Modelscited by 4
  20. 2026TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignmentcited by 4
  21. 2026Z-Reward: Beyond Scalar Rewards by Internalizing Reasoning into Score Distributionscited by 4
  22. 2026Thinking with Frames: Generative Video Distortion Evaluation via Frame Reward Modelcited by 3
  23. 2026Joint Reward Modeling: Internalizing Chain-of-Thought for Efficient Visual Reward Modelscited by 3
  24. 2026SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuningcited by 3
  25. 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
  26. 2026AesRM: Improving Video Aesthetics with Expert-Level Feedbackcited by 3
  27. 2026Are we really tilting? The mechanics of reward guidance in flow and diffusion modelscited by 3
  28. 2026Advances in GRPO for Generation Models: A Surveycited by 2
  29. 2026Reward Sharpness-Aware Fine-Tuning for Diffusion Modelscited by 2
  30. 2026PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learningcited by 2
  31. 2026ViPO: Visual Preference Optimization at Scalecited by 2
  32. 2026Think, then Score: Decoupled Reasoning and Scoring for Video Reward Modelingcited by 2
  33. 2026OmniVerifier-M1: Multimodal Meta-Verifier with Explicit Structured Recalibrationcited by 2
  34. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2
  35. 2025VLIC: Vision-Language Models As Perceptual Judges for Human-Aligned Image Compressioncited by 1
  36. 2026AdvDMD: Adversarial Reward Meets DMD For High-Quality Few-Step Generationcited by 1
  37. 2026StressDream: Steering Video World Models for Robust Policy Evaluation and Improvementcited by 1
  38. 2026Improving Visual Representation Alignment Generation with GRPOcited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimizing Visual Generative Models via Distribution-wise Rewards
  2. 2026Aligning One-Step Generative Models with Reward-Weighted Transport Distillation
  3. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
  4. 2026Think Before You Score: Thinking Reward Model for Visual Generation
  5. 2025The Image as Its Own Reward: Reinforcement Learning with Adversarial Reward for Image Generation
  6. 2026Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
  7. 2026Visual-ERM: Reward Modeling for Visual Equivalence
  8. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model