RealDPO: Real or Not Real, that is the Preference
On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment
Abstract (arXiv)
Video generative models have recently achieved notable advancements in synthesis quality. However, generating complex motions remains a critical challenge, as existing models often struggle to produce natural, smooth, and contextually consistent movements. This gap between generated and real-world motions limits their practical applicability. To address this issue, we introduce RealDPO, a novel alignment paradigm that leverages real-world data as positive samples for preference learning, enabling more accurate motion synthesis. Unlike traditional supervised fine-tuning (SFT), which offers limited corrective feedback, RealDPO employs Direct Preference Optimization (DPO) with a tailored loss function to enhance motion realism. By contrasting real-world videos with erroneous model outputs, RealDPO enables iterative self-correction, progressively refining motion quality. To support post-training in complex motion synthesis, we propose RealAction-5K, a curated dataset of high-quality videos capturing human daily activities with rich and precise motion details. Extensive experiments demonstrate that RealDPO significantly improves video quality, text alignment, and motion realism compared to state-of-the-art models and existing preference optimization techniques.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 38 references that are in the map. First posted on arXiv 16 Oct 2025.
Builds on
34 of 38 references.
Closest ancestors · 17
Same direction, newest first
- 2025Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation14 Aug
- 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models4 Jun
- 2025Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences2 Jun
- 2025Self-NPO: Data-Free Diffusion Model Enhancement via Truncated Diffusion Fine-Tuning17 May
- 2025VPO: Aligning Text-to-Video Generation Models with Prompt Optimization26 Mar
- 2025Unified Reward Model for Multimodal Understanding and Generation7 Mar
- 2025Improving Video Generation with Human Feedback23 Jan
- 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation30 Dec
- 2024OnlineVPO: Align Video Diffusion Model with Online Video-Centric Preference Optimization19 Dec
- 2024VideoDPO: Omni-Preference Alignment for Video Diffusion Generation18 Dec
- 2024LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment6 Dec
- 2023InstructVideo: Instructing Video Diffusion Models with Human Feedback19 Dec
5 more
- 2023Rich Human Feedback for Text-to-Image Generation15 Dec
- 2023Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model22 Nov
- 2023Diffusion Model Alignment Using Direct Preference Optimization21 Nov
- 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewards29 Sep
- 2023Aligning Text-to-Image Models using Human Feedback23 Feb
Other references · 17
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
- 2023Training Diffusion Models with Reinforcement Learningcited by 982
- 2023ModelScope Text-to-Video Technical Reportcited by 658
- 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
- 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
- 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
5 more
- 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 350
- 2024Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understandingcited by 298
- 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
- 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
- 2025SwiftVideo: A Unified Framework for Few-Step Video Generation through Trajectory-Distribution Alignmentcited by 10
Built on by
Citing papers per year: 2025: 1; 2026: 5.
6 papers.
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
- 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
- 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuningcited by 10
- 2026VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoningcited by 6
- 2026NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learningcited by 0
- 2026Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusioncited by 0
- 2026VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning26 Aug 2026
- 2026Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion30 Jul 2026
- 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligence8 Jul 2026
- 2026NormGuard: Reward-Preserving Norm Constraints in Flow-Matching Reinforcement Learning26 Jun 2026
- 2026Diffusion-DRF: Free, Rich, and Differentiable Reward for Video Diffusion Fine-Tuning7 Jan 2026
- 2025Simulating the Visual World with Artificial Intelligence: A Roadmap11 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2024SoPo: Text-to-Motion Generation Using Semi-Online Preference Optimization
- 2026Mind the Generative Details: Direct Localized Detail Preference Optimization for Video Diffusion Models
- 2024MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization
- 2025AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animation
- 2025Discriminator-Free Direct Preference Optimization for Video Diffusion
- 2026DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data
- 2026PhyCo: Learning Controllable Physical Priors for Generative Motion
- 2026Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers