mlsanityData to 7 Oct 2026

DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory

arXiv 2308.08089 ↗First posted 16 Aug 2023cs.CVcited by 260

Not on the map. Why?

Cited by 102 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Controllable video generation has gained significant attention in recent years. However, two main limitations persist: Firstly, most existing works focus on either text, image, or trajectory-based control, leading to an inability to achieve fine-grained control in videos. Secondly, trajectory control research is still in its early stages, with most experiments being conducted on simple datasets like Human3.6M. This constraint limits the models' capability to process open-domain images and effectively handle complex curved trajectories. In this paper, we propose DragNUWA, an open-domain diffusion-based video generation model. To tackle the issue of insufficient control granularity in existing works, we simultaneously introduce text, image, and trajectory information to provide fine-grained control over video content from semantic, spatial, and temporal perspectives. To resolve the problem of limited open-domain trajectory control in current research, We propose trajectory modeling with three aspects: a Trajectory Sampler (TS) to enable open-domain control of arbitrary trajectories, a Multiscale Fusion (MF) to control trajectories in different granularities, and an Adaptive Training (AT) strategy to generate consistent videos following trajectories. Our experiments validate the effectiveness of DragNUWA, demonstrating its superior performance in fine-grained control in video generation. The homepage link is urlhttps://www.microsoft.com/en-us/research/project/dragnuwa/

Key facts. Cited by 260 papers in the map (data through 7 Oct 2026). Builds on 29 of its 31 references that are in the map. First posted on arXiv 16 Aug 2023.

Builds on

29 of 31 references.

References · 29

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  6. 2019Deep High-Resolution Representation Learning for Visual Recognitioncited by 1,417
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  9. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  10. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  11. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  12. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
17 more
  1. 2018Everybody Dance Nowcited by 313
  2. 2022Unifying Flow, Stereo and Depth Estimationcited by 270
  3. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  4. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  5. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  6. 2019Few-shot Video-to-Video Synthesiscited by 196
  7. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  8. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  9. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  10. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  11. 2017Dual Motion GAN for Future-Flow Embedded Video Predictioncited by 69
  12. 2017Recurrent Environment Simulatorscited by 68
  13. 2020Future Video Synthesis with Object Motion Predictioncited by 57
  14. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  15. 2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
  16. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  17. 2021Click to Move: Controlling Video Generation with Sparse Motioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 81; 2025: 96; 2026: 62.

260 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  3. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  5. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  6. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  7. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  8. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  9. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  10. 2023A Survey on Video Diffusion Modelscited by 203
  11. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  12. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
38 more
  1. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  2. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  3. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  4. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  5. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  6. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  7. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  8. 2024Training-free Camera Control for Video Generationcited by 119
  9. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  10. 2025WorldModelBench: Judging Video Generation Models As World Modelscited by 116
  11. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  12. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  13. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  14. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  15. 2023Generative Image Dynamicscited by 93
  16. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 93
  17. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 92
  18. 2023VideoLCM: Video Latent Consistency Modelcited by 90
  19. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  20. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
  21. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  22. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  23. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  24. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  25. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  26. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  27. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
  28. 2024HumanVid: Demystifying Training Data for Camera-controllable Human Image Animationcited by 63
  29. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  30. 2024VideoStudio: Generating Consistent-Content and Multi-Scene Videoscited by 62
  31. 2024Image Conductor: Precision Control for Interactive Video Synthesiscited by 62
  32. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 60
  33. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  34. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Modelscited by 55
  35. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 55
  36. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  37. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  38. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51

50 most cited of 260 All 260 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships
  2. 2024Trajectory Attention for Fine-grained Video Motion Control
  3. 2025DiTraj: training-free trajectory control for video diffusion transformer
  4. 2024Mojito: Motion Trajectory and Intensity Control for Video Generation
  5. 2023DragVideo: Interactive Drag-style Video Editing
  6. 2025Controllable Video Generation: A Survey
  7. 2024Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion
  8. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generation