# DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

arXiv: https://arxiv.org/abs/2506.03517 · HTML: https://mlsanity.com/paper/2506.03517/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 86 of its 91 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 30 of 86)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow](https://mlsanity.com/paper/2209.03003/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Training Diffusion Models with Reinforcement Learning](https://mlsanity.com/paper/2305.13301/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [Diffusion Model Alignment Using Direct Preference Optimization](https://mlsanity.com/paper/2311.12908/) (2023)
- [RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment](https://mlsanity.com/paper/2304.06767/) (2023)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)

## Built on by (most cited first, 30 of 34)

- [JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation](https://mlsanity.com/paper/2602.19163/) (2026)
- [PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation](https://mlsanity.com/paper/2512.24551/) (2025)
- [Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances](https://mlsanity.com/paper/2508.10316/) (2025)
- [Identity-GRPO: Optimizing Multi-Human Identity-preserving Video Generation via Reinforcement Learning](https://mlsanity.com/paper/2510.14256/) (2025)
- [Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuning](https://mlsanity.com/paper/2601.04153/) (2026)
- [WorldPack: Dynamic Frame Compression for Long-context Video World Modeling](https://mlsanity.com/paper/2512.02473/) (2025)
- [The Pulse of Motion: Measuring Physical Frame Rate from Visual Dynamics](https://mlsanity.com/paper/2603.14375/) (2026)
- [Towards Better Optimization For Listwise Preference in Diffusion Models](https://mlsanity.com/paper/2510.01540/) (2025)
- [TeleBoost: A Systematic Alignment Framework for High-Fidelity, Controllable, and Robust Video Generation](https://mlsanity.com/paper/2602.07595/) (2026)
- [RealDPO: Real or Not Real, that is the Preference](https://mlsanity.com/paper/2510.14955/) (2025)
- [ID-Crafter: VLM-Grounded Online RL for Compositional Multi-Subject Video Generation](https://mlsanity.com/paper/2511.00511/) (2025)
- [GlyphPrinter: Region-Grouped Direct Preference Optimization for Glyph-Accurate Visual Text Rendering](https://mlsanity.com/paper/2603.15616/) (2026)
- [HP-Edit: A Human-Preference Post-Training Framework for Image Editing](https://mlsanity.com/paper/2604.19406/) (2026)
- [Diverse Video Generation with Determinantal Point Process-Guided Policy Optimization](https://mlsanity.com/paper/2511.20647/) (2025)
- [SoliReward: Mitigating Susceptibility to Reward Hacking and Annotation Noise in Video Generation Reward Models](https://mlsanity.com/paper/2512.22170/) (2025)
- [Mind the Generative Details: Direct Localized Detail Preference Optimization for Video Diffusion Models](https://mlsanity.com/paper/2601.04068/) (2026)
- [VEFX-Bench: A Holistic Benchmark for Generic Video Editing and Visual Effects](https://mlsanity.com/paper/2604.16272/) (2026)
- [When Physical Preferences Meet Semantic Constraints: Physical and Semantic Direct Preference Optimization for Text-to-Video Generation](https://mlsanity.com/paper/2607.16947/) (2026)
- [SHIFT: Motion Alignment in Video Diffusion Models with Adversarial Hybrid Fine-Tuning](https://mlsanity.com/paper/2603.17426/) (2026)
- [Reg-DPO: SFT-Regularized Direct Preference Optimization with GT-Pair for Improving Video Generation](https://mlsanity.com/paper/2511.01450/) (2025)
- [CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL](https://mlsanity.com/paper/2605.14274/) (2026)
- [DeDPO: Debiased Direct Preference Optimization for Diffusion Models](https://mlsanity.com/paper/2602.06195/) (2026)
- [Risk-Controllable Multi-View Diffusion for Driving Scenario Generation](https://mlsanity.com/paper/2603.11534/) (2026)
- [Place-it-R1: Unlocking Environment-aware Reasoning Potential of MLLM for Video Object Insertion](https://mlsanity.com/paper/2603.06140/) (2026)
- [GDPO-Listener: Expressive Interactive Head Generation via Auto-Regressive Flow Matching and Group reward-Decoupled Policy Optimization](https://mlsanity.com/paper/2603.25020/) (2026)
- [Explicit Critic Guidance for Aligning Diffusion Models](https://mlsanity.com/paper/2605.27736/) (2026)
- [Can We Predict The Human Preference For Text-to-Image Content Prior To Generation And Is It Even Useful To Do So?](https://mlsanity.com/paper/2606.05478/) (2026)
- [Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion](https://mlsanity.com/paper/2607.28058/) (2026)
- [Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking](https://mlsanity.com/paper/2608.20011/) (2026)
- [Aligning Human Sense: Calibrated Distributional Reward Learning for Video Generation](https://mlsanity.com/paper/2608.21425/) (2026)

## Abstract (arXiv)

Direct Preference Optimization (DPO) has recently been applied as a post-training technique for text-to-video diffusion models. To obtain training data, annotators are asked to provide preferences between two videos generated from independent noise. However, this approach prohibits fine-grained comparisons, and we point out that it biases the annotators towards low-motion clips as they often contain fewer visual artifacts. In this work, we introduce DenseDPO, a method that addresses these shortcomings by making three contributions. First, we create each video pair for DPO by denoising corrupted copies of a ground truth video. This results in aligned pairs with similar motion structures while differing in local details, effectively neutralizing the motion bias. Second, we leverage the resulting temporal alignment to label preferences on short segments rather than entire clips, yielding a denser and more precise learning signal. With only one-third of the labeled data, DenseDPO greatly improves motion generation over vanilla DPO, while matching it in text alignment, visual quality, and temporal consistency. Finally, we show that DenseDPO unlocks automatic preference annotation using off-the-shelf Vision Language Models (VLMs): GPT accurately predicts segment-level preferences similar to task-specifically fine-tuned video reward models, and DenseDPO trained on these labels achieves performance close to using human labels.