mlsanityData to 14 Oct 2026

Video Diffusion Alignment via Reward Gradients

arXiv 2407.08737 ↗First posted 11 Jul 2024cs.CV cs.AI cs.LGcited by 107

On the map: Diffusion models → Video generation

also in 1 more
Abstract (arXiv)

We have made significant progress towards building foundational video diffusion models. As these models are trained using large-scale unsupervised data, it has become crucial to adapt these models to specific downstream tasks. Adapting these models via supervised fine-tuning requires collecting target datasets of videos, which is challenging and tedious. In this work, we utilize pre-trained reward models that are learned via preferences on top of powerful vision discriminative models to adapt video diffusion models. These models contain dense gradient information with respect to generated RGB pixels, which is critical to efficient learning in complex search spaces, such as videos. We show that backpropagating gradients from these reward models to a video diffusion model can allow for compute and sample efficient alignment of the video diffusion model. We show results across a variety of reward models and video diffusion models, demonstrating that our approach can learn much more efficiently in terms of reward queries and computation than prior gradient-free approaches. Our code, model weights,and more visualization are available at https://vader-vid.github.io.

Key facts. Cited by 107 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 30 of its 34 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

30 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  2. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedback19 Dec
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  4. 2023ModelScope Text-to-Video Technical Report12 Aug
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  9. 2017The Kinetics Human Action Video Datasetcited by 1,909
  10. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,489
  11. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
  12. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
13 more
  1. 2023Training Diffusion Models with Reinforcement Learningcited by 982
  2. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  3. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  4. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  8. 2023Aligning Text-to-Image Models using Human Feedbackcited by 422
  9. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  10. 2023Aligning Text-to-Image Diffusion Models with Reward Backpropagationcited by 288
  11. 2024Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillationcited by 279
  12. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  13. 2017Unbiasing Truncated Backpropagation Through Timecited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 49; 2026: 44.

107 papers.

Sort:

By area:

  1. 2025Flow-GRPO: Training Flow Matching Models via Online RLcited by 661
  2. 2025DanceGRPO: Unleashing GRPO on Visual Generationcited by 413
  3. 2025Improving Video Generation with Human Feedbackcited by 284
  4. 2025DiffusionNFT: Online Diffusion Reinforcement with Forward Processcited by 230
  5. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 124
  6. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  7. 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generationcited by 108
  8. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 86
  9. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  10. 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignmentcited by 53
  11. 2025PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Dropcited by 50
  12. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
38 more
  1. 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimizationcited by 41
  2. 2025Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Searchcited by 41
  3. 2025Scaling Image and Video Generation via Test-Time Evolutionary Searchcited by 40
  4. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 35
  5. 2025Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Modelscited by 35
  6. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 34
  7. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  8. 2024The Dawn of Video Generation: Preliminary Explorations with SORA-like Modelscited by 31
  9. 2024StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaborationcited by 31
  10. 2025Directly Aligning the Full Diffusion Trajectory with Fine-Grained Human Preferencecited by 30
  11. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  12. 2024DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimizationcited by 26
  13. 2025VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learningcited by 26
  14. 2025DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundnesscited by 25
  15. 2025Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Modelscited by 23
  16. 2025InPO: Inversion Preference Optimization with Reparametrized DDIM for Efficient Diffusion Model Alignmentcited by 19
  17. 2025Taming Preference Mode Collapse via Directional Decoupling Alignment in Diffusion Reinforcement Learningcited by 18
  18. 2025MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generationcited by 16
  19. 2025PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selectioncited by 15
  20. 2025Dual-IPO: Dual-Iterative Preference Optimization for Text-to-Video Generationcited by 13
  21. 2025Smoothed Preference Optimization via ReNoise Inversion for Aligning Diffusion Models with Varied Human Preferencescited by 13
  22. 2026Unified Personalized Reward Model for Vision Generationcited by 13
  23. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generationcited by 13
  24. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learningcited by 12
  25. 2025Discrete Diffusion Trajectory Alignment via Stepwise Decompositioncited by 12
  26. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advancescited by 12
  27. 2025PhysMaster: Mastering Physical Representation for Video Generation via Reinforcement Learningcited by 12
  28. 2025Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generatorcited by 11
  29. 2025Composition and Alignment of Diffusion Models using Constrained Learningcited by 10
  30. 2025Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learningcited by 10
  31. 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuningcited by 10
  32. 2026TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignmentcited by 10
  33. 2024Free2Guide: Training-Free Text-to-Video Alignment using Image LVLMcited by 9
  34. 2025RDPO: Real Data Preference Optimization for Physics Consistency Video Generationcited by 9
  35. 2025Real-Time Motion-Controllable Autoregressive Video Diffusioncited by 9
  36. 2025ImagerySearch: Adaptive Test-Time Search for Video Generation Beyond Semantic Dependency Constraintscited by 9
  37. 2025PSR: Scaling Multi-Subject Personalized Image Generation with Pairwise Subject-Consistency Rewardscited by 9
  38. 2025Identity-Preserving Image-to-Video Generation via Reward-Guided Optimizationcited by 8

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Lightning: Fast Video Generation via Homologous Preference Distillation
  2. 2026Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion
  3. 2026Beyond VLM-Based Rewards: Diffusion-Native Latent Reward Modeling
  4. 2026VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment
  5. 2025Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Models
  6. 2025Goal-Driven Reward by Video Diffusion Models for Reinforcement Learning
  7. 2025Alignment and Safety of Diffusion Models via Reinforcement Learning and Reward Modeling: A Survey
  8. 2026G3-LoRA: Organizing Reward-Weighted Video Data with Gradient-Guided Grouped LoRA