mlsanityData to 7 Oct 2026

Motion-Conditioned Diffusion Model for Controllable Video Synthesis

arXiv 2304.14404 ↗First posted 27 Apr 2023cs.CVcited by 99

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent advancements in diffusion models have greatly improved the quality and diversity of synthesized content. To harness the expressive power of diffusion models, researchers have explored various controllable mechanisms that allow users to intuitively guide the content synthesis process. Although the latest efforts have primarily focused on video synthesis, there has been a lack of effective methods for controlling and describing desired content and motion. In response to this gap, we introduce MCDiff, a conditional diffusion model that generates a video from a starting image frame and a set of strokes, which allow users to specify the intended content and dynamics for synthesis. To tackle the ambiguity of sparse motion inputs and achieve better synthesis quality, MCDiff first utilizes a flow completion model to predict the dense video motion based on the semantic understanding of the video frame and the sparse motion control. Then, the diffusion model synthesizes high-quality future frames to form the output video. We qualitatively and quantitatively show that MCDiff achieves the state-the-of-art visual quality in stroke-guided controllable video synthesis. Additional experiments on MPII Human Pose further exhibit the capability of our model on diverse content and motion synthesis.

Key facts. Cited by 99 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 48 of its 55 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

48 of 55 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  4. 2022SinFusion: Training Diffusion Models on a Single Image or Video21 Nov
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  6. 2022Diffusion Models for Video Prediction and Infilling15 Jun
  7. 2022Flexible Diffusion Modeling of Long Videos23 May
  8. 2022Video Diffusion Models7 Apr
  9. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
27 more
  1. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  2. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  3. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  5. 2020First Order Motion Model for Image Animationcited by 698
  6. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  7. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  8. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  9. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  10. 2021Motion Representations for Articulated Animationcited by 253
  11. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  12. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 206
  13. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  14. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  15. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  16. 2020Stochastic Latent Residual Video Predictioncited by 106
  17. 2019Unsupervised Learning of Object Structure and Dynamics from Videoscited by 92
  18. 2020Semantic Image Manipulation Using Scene Graphscited by 82
  19. 2021Playable Video Generationcited by 63
  20. 2022Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Trainingcited by 56
  21. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  22. 2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
  23. 2022Diffusion Video Autoencoders: Toward Temporally Consistent Face Video Editing via Disentangled Video Encodingcited by 39
  24. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  25. 2022Playable Environments: Video Manipulation in Space and Timecited by 29
  26. 20233D-aware Conditional Image Synthesiscited by 24
  27. 2021Click to Move: Controlling Video Generation with Sparse Motioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 38; 2025: 30; 2026: 16.

99 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  3. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  4. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  5. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  6. 2023A Survey on Video Diffusion Modelscited by 203
  7. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  8. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  9. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 179
  10. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  11. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  12. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
38 more
  1. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  2. 2024Training-free Camera Control for Video Generationcited by 119
  3. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  4. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  5. 2023Generative Image Dynamicscited by 93
  6. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 93
  7. 2025Controllable Video Generation: A Surveycited by 93
  8. 2023Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Taskcited by 91
  9. 2023VideoLCM: Video Latent Consistency Modelcited by 90
  10. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  11. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  12. 2025Multi-subject Open-set Personalization in Video Generationcited by 65
  13. 2024360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Modelcited by 56
  14. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 55
  15. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51
  16. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 51
  17. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 47
  18. 2024Foundation Models for Video Understanding: A Surveycited by 46
  19. 2023LivePhoto: Real Image Animation with Text-guided Motion Controlcited by 44
  20. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  21. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  22. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  23. 2024Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimatorscited by 39
  24. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  25. 2025FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesiscited by 37
  26. 2024DragAPart: Learning a Part-Level Motion Prior for Articulated Objectscited by 32
  27. 2023SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Processcited by 28
  28. 2024Video Diffusion Models: A Surveycited by 28
  29. 2024Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamicscited by 28
  30. 2025Generative Physical AI in Vision: A Surveycited by 24
  31. 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understandingcited by 22
  32. 2025Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Frameworkcited by 20
  33. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  34. 2024ObjCtrl-2.5D: Training-free Object Control with Camera Posescited by 18
  35. 2025OmniVCus: Feedforward Subject-driven Video Customization with Multimodal Control Conditionscited by 17
  36. 2024MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Modelscited by 16
  37. 2023MoVideo: Motion-Aware Video Generation with Diffusion Modelscited by 15
  38. 2024OnlyFlow: Optical Flow based Motion Conditioning for Video Diffusion Modelscited by 15

50 most cited of 99 All 99 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learning
  2. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  3. 2023VideoControlNet: A Motion-Guided Video-to-Video Translation Framework by Using Diffusion Model with ControlNet
  4. 2026Efficient Video Diffusion Models: Advancements and Challenges
  5. 2024Controllable Longer Image Animation with Diffusion Models
  6. 2024Towards motion from video diffusion models
  7. 2023BIVDiff: A Training-Free Framework for General-Purpose Video Synthesis via Bridging Image and Video Diffusion Models
  8. 2023TrackDiffusion: Tracklet-Conditioned Video Generation via Diffusion Models