mlsanityData to 7 Oct 2026

MotionCtrl: A Unified and Flexible Motion Controller for Video Generation

arXiv 2312.03641 ↗First posted 6 Dec 2023cs.CV cs.AI cs.LGcited by 568

Not on the map. Why?

Cited by 183 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. Accurate control of both camera and object motion is essential for video generation. However, existing works either mainly focus on one type of motion or do not clearly distinguish between the two, limiting their control capabilities and diversity. Therefore, this paper presents MotionCtrl, a unified and flexible motion controller for video generation designed to effectively and independently control camera and object motion. The architecture and training strategy of MotionCtrl are carefully devised, taking into account the inherent properties of camera motion, object motion, and imperfect training data. Compared to previous methods, MotionCtrl offers three main advantages: 1) It effectively and independently controls camera motion and object motion, enabling more fine-grained motion control and facilitating flexible and diverse combinations of both types of motion. 2) Its motion conditions are determined by camera poses and trajectories, which are appearance-free and minimally impact the appearance or shape of objects in generated videos. 3) It is a relatively generalizable model that can adapt to a wide array of camera poses and trajectories once trained. Extensive qualitative and quantitative experiments have been conducted to demonstrate the superiority of MotionCtrl over existing methods. Project Page: https://wzhouxiff.github.io/projects/MotionCtrl/

Key facts. Cited by 568 papers in the map (data through 7 Oct 2026). Builds on 39 of its 44 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

39 of 44 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
27 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  6. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  10. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  11. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  12. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  13. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  14. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  15. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  16. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  17. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  18. 2016Generating Videos with Scene Dynamicscited by 376
  19. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  20. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  21. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  22. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  23. 2019Few-shot Video-to-Video Synthesiscited by 196
  24. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  25. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesiscited by 99
  26. 2022ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wildcited by 90
  27. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generationcited by 59

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 113; 2025: 240; 2026: 213.

568 papers.

Sort:

By area:

  1. 2025VACE: All-in-One Video Creation and Editingcited by 467
  2. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  3. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  4. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  5. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  6. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  7. 2023PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embeddingcited by 324
  8. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  9. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  10. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  11. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 179
  12. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
38 more
  1. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  2. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  3. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  4. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  5. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  6. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  7. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  8. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  9. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  10. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  11. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  12. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  13. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 139
  14. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  15. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 129
  16. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  17. 2025Video World Models with Long-term Spatial Memorycited by 124
  18. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  19. 2024Training-free Camera Control for Video Generationcited by 119
  20. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  21. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  22. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  23. 2024MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Viewscited by 111
  24. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  25. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  26. 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 102
  27. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  28. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  29. 2025Aether: Geometric-Aware Unified World Modelingcited by 94
  30. 2025Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generationcited by 93
  31. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  32. 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 92
  33. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generationcited by 90
  34. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
  35. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  36. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  37. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  38. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizercited by 79

50 most cited of 568 All 568 (JSON)