mlsanityData to 14 Oct 2026

Improving Video Generation with Human Feedback

arXiv 2501.13918 ↗First posted 23 Jan 2025cs.CV cs.AI cs.GRcited by 284

On the map: Diffusion models → Video generation

also in 1 more
Abstract (arXiv)

Video generation has achieved significant advances through rectified flow techniques, but issues like unsmooth motion and misalignment between videos and prompts persist. In this work, we develop a systematic pipeline that harnesses human feedback to mitigate these problems and refine the video generation model. Specifically, we begin by constructing a large-scale human preference dataset focused on modern video generation models, incorporating pairwise annotations across multi-dimensions. We then introduce VideoReward, a multi-dimensional video reward model, and examine how annotations and various design choices impact its rewarding efficacy. From a unified reinforcement learning perspective aimed at maximizing reward with KL regularization, we introduce three alignment algorithms for flow-based models. These include two training-time strategies: direct preference optimization for flow (Flow-DPO) and reward weighted regression for flow (Flow-RWR), and an inference-time technique, Flow-NRG, which applies reward guidance directly to noisy videos. Experimental results indicate that VideoReward significantly outperforms existing reward models, and Flow-DPO demonstrates superior performance compared to both Flow-RWR and supervised fine-tuning methods. Additionally, Flow-NRG lets users assign custom weights to multiple objectives during inference, meeting personalized video quality needs.

Key facts. Cited by 284 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 70 of its 89 references that are in the map. First posted on arXiv 23 Jan 2025.

Builds on

70 of 89 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimization19 Dec
  2. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generation18 Dec
  3. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Design8 Oct
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  5. 2024Video Diffusion Alignment via Reward Gradients11 Jul
  6. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedback29 May
  7. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  8. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedback19 Dec
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov

Other references · 61

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2017Proximal Policy Optimization Algorithmscited by 15,215
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  12. 2022Scalable Diffusion Models with Transformerscited by 6,054
38 more
  1. 2022Flow Matching for Generative Modelingcited by 5,702
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  5. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  6. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  8. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  9. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  10. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  11. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  12. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  14. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  15. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  16. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 652
  17. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  18. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  19. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  20. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  21. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 340
  22. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 294
  23. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  24. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 287
  25. 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Modelcited by 252
  26. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  27. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  28. 2024Adjoint Matching: Fine-tuning Flow and Diffusion Generative Models with Memoryless Stochastic Optimal Controlcited by 233
  29. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  30. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  31. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  32. 2023Generative Judge for Evaluating Alignmentcited by 169
  33. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 156
  34. 2023Rich Human Feedback for Text-to-Image Generationcited by 141
  35. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 130
  36. 2024Learning Multi-dimensional Human Preference for Text-to-Image Generationcited by 106
  37. 2023Guided Flows for Generative Modeling and Decision Makingcited by 101
  38. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 90

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 105; 2026: 179.

284 papers.

Sort:

By area:

  1. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  2. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  3. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 259
  4. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 229
  5. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  6. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  7. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  8. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 192
  9. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  10. 2025MixGRPO: Unlocking Flow-based GRPO Efficiency with Mixed ODE-SDEcited by 178
  11. 2025Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuningcited by 90
  12. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 86
38 more
  1. 2025LongCat-Video Technical Reportcited by 85
  2. 2025RewardDance: Reward Scaling in Visual Generationcited by 68
  3. 2026Helios: Real Real-Time Long Video Generation Modelcited by 65
  4. 2025EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editingcited by 62
  5. 2025BranchGRPO: Stable and Efficient GRPO with Structured Branching in Diffusion Modelscited by 61
  6. 2025From Reflection to Perfection: Scaling Inference-Time Optimization for Text-to-Image Diffusion Models via Reflection Tuningcited by 56
  7. 2025Distribution Matching Distillation Meets Reinforcement Learningcited by 50
  8. 2025VMBench: A Benchmark for Perception-Aligned Video Motion Generationcited by 49
  9. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 49
  10. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 40
  11. 2025VideoScore2: Think before You Score in Generative Video Evaluationcited by 40
  12. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  13. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  14. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  15. 2025Epipolar Geometry Improves Video Generation Modelscited by 26
  16. 2025DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundnesscited by 25
  17. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
  18. 2026VGGRPO: Towards World-Consistent Video Generation with 4D Latent Rewardcited by 24
  19. 2025VideoGen-Eval: Agent-based System for Video Generation Evaluationcited by 23
  20. 2025OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learningcited by 23
  21. 2025PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generationcited by 23
  22. 2026VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generationcited by 23
  23. 2026Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stackcited by 22
  24. 2025Follow-Your-Preference: Towards Preference-Aligned Image Inpaintingcited by 18
  25. 2025Phantom-Data : Towards a General Subject-Consistent Video Generation Datasetcited by 17
  26. 2025Learning Human-Perceived Fakeness in AI-Generated Videos via Multimodal LLMscited by 17
  27. 2025GARDO: Reinforcing Diffusion Models without Reward Hackingcited by 17
  28. 2025Hallo4: High-Fidelity Dynamic Portrait Animation via Direct Preference Optimizationcited by 16
  29. 2025Video Generation Models Are Good Latent Reward Modelscited by 16
  30. 2025VIVA: VLM-Guided Instruction-Based Video Editing with Reward Optimizationcited by 16
  31. 2025InfLVG: Reinforce Inference-Time Consistent Long Video Generation with GRPOcited by 15
  32. 2025Reinforcement Learning for Flow-Matching Policiescited by 15
  33. 2025PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selectioncited by 15
  34. 2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
  35. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  36. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Followingcited by 15
  37. 2026Self-Refining Video Samplingcited by 15
  38. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generationcited by 15

50 most cited of 284 All 284 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation
  2. 2025HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment
  3. 2026Advances in GRPO for Generation Models: A Survey
  4. 2024Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences
  5. 2024MotionRL: Align Text-to-Motion Generation to Human Preferences with Multi-Reward Reinforcement Learning
  6. 2026CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models
  7. 2025Harness Local Rewards for Global Benefits: Effective Text-to-Video Generation Alignment with Patch-level Reward Models
  8. 2025GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning