mlsanityData to 14 Oct 2026

Motion Prompting: Controlling Video Generation with Motion Trajectories

arXiv 2412.02700 ↗First posted 3 Dec 2024cs.CVcited by 178

Not on the map. Why?

Cited by 51 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Motion control is crucial for generating expressive and compelling video content; however, most existing video generation models rely mainly on text prompts for control, which struggle to capture the nuances of dynamic actions and temporal compositions. To this end, we train a video generation model conditioned on spatio-temporally sparse or dense motion trajectories. In contrast to prior motion conditioning work, this flexible representation can encode any number of trajectories, object-specific or global scene motion, and temporally sparse motion; due to its flexibility we refer to this conditioning as motion prompts. While users may directly specify sparse trajectories, we also show how to translate high-level user requests into detailed, semi-dense motion prompts, a process we term motion prompt expansion. We demonstrate the versatility of our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing. Our results showcase emergent behaviors, such as realistic physics, suggesting the potential of motion prompts for probing video models and interacting with future generative world models. Finally, we evaluate quantitatively, conduct a human study, and demonstrate strong performance. Video results are available on our webpage: https://motion-prompting.github.io/

Key facts. Cited by 178 papers in the map (data through 14 Oct 2026). Builds on 76 of its 94 references that are in the map. First posted on arXiv 3 Dec 2024.

Builds on

76 of 94 references.

References · 76

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2023Segment Anythingcited by 8,234
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  11. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  12. 2022Video Diffusion Modelscited by 2,305
38 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  4. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,059
  5. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 968
  6. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  8. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  9. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  10. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  11. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  12. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  13. 2023CoTracker: It is Better to Track Togethercited by 522
  14. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  15. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  16. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  17. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  18. 2024UniDepth: Universal Monocular Metric Depth Estimationcited by 382
  19. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 368
  20. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  21. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 294
  22. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 287
  23. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 276
  24. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  25. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 255
  26. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  27. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 219
  28. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 217
  29. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 212
  30. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 211
  31. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 195
  32. 2021COTR: Correspondence Transformer for Matching Across Imagescited by 183
  33. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 174
  34. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 173
  35. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 164
  36. 2024DragAnything: Motion Control for Anything using Entity Representationcited by 159
  37. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 159
  38. 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 120

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 81; 2026: 97.

178 papers.

Sort:

By area:

  1. 2025Video models are zero-shot learners and reasonerscited by 220
  2. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 185
  3. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 117
  4. 2025Controllable Video Generation: A Surveycited by 98
  5. 2025Aether: Geometric-Aware Unified World Modelingcited by 97
  6. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  7. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 77
  8. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 76
  9. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 61
  10. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 53
  11. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  12. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 47
38 more
  1. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 45
  2. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  3. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 43
  4. 2025PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generationcited by 41
  5. 2025MultiShotMaster: A Controllable Multi-Shot Video Generation Frameworkcited by 36
  6. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  7. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 34
  8. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  9. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 31
  10. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  11. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  12. 2025Generative Physical AI in Vision: A Surveycited by 24
  13. 2025Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representationcited by 23
  14. 2025Generative Video Motion Editing with 3D Point Trackscited by 23
  15. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 22
  16. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulationcited by 17
  17. 2025MoAlign: Motion-Centric Representation Alignment for Video Diffusion Modelscited by 16
  18. 2025Synthetic Video Enhances Physical Fidelity in Video Synthesiscited by 15
  19. 2025Precise Action-to-Video Generation Through Visual Action Promptscited by 15
  20. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  21. 2025Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfercited by 14
  22. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  23. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 14
  24. 2025MotionV2V: Editing Motion in a Videocited by 14
  25. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 14
  26. 2026Physical Simulator In-the-Loop Video Generationcited by 14
  27. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  28. 2026LOME: Learning Human-Object Manipulation with Action-Conditioned Egocentric World Modelcited by 13
  29. 2025VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learningcited by 12
  30. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  31. 2025Frame Guidance: Training-Free Guidance for Frame-Level Control in Video Diffusion Modelscited by 11
  32. 2025DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generationcited by 11
  33. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  34. 2025AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Modelscited by 10
  35. 2025Mask2IV: Interaction-Centric Video Generation via Mask Trajectoriescited by 10
  36. 2026SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generationcited by 10
  37. 2025MASIV: Toward Material-Agnostic System Identification from Videoscited by 9
  38. 2025Lynx: Towards High-Fidelity Personalized Video Generationcited by 9

50 most cited of 178 All 178 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  2. 2024Move-in-2D: 2D-Conditioned Human Motion Generation
  3. 2025LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer
  4. 2023Make Pixels Dance: High-Dynamic Video Generation
  5. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generation
  6. 2024Optical-Flow Guided Prompt Optimization for Coherent Video Generation
  7. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion
  8. 2024Disentangling Foreground and Background Motion for Enhanced Realism in Human Video Generation