mlsanityData to 7 Oct 2026

Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

arXiv 2401.15977 ↗First posted 29 Jan 2024cs.CVcited by 209

Not on the map. Why?

Cited by 71 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Motion-I2V, a novel framework for consistent and controllable image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusion-based motion field predictor, which focuses on deducing the trajectories of the reference image's pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image's feature to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even at the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region annotations. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V's second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation. Please see our project page at https://xiaoyushi97.github.io/Motion-I2V/.

Key facts. Cited by 209 papers in the map (data through 7 Oct 2026). Builds on 64 of its 88 references that are in the map. First posted on arXiv 29 Jan 2024.

Builds on

64 of 88 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
38 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  4. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  5. 2015FlowNet: Learning Optical Flow with Convolutional Networkscited by 1,054
  6. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  7. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,021
  8. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  9. 2017PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volumecited by 966
  10. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  11. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  12. 2023CoTracker: It is Better to Track Togethercited by 516
  13. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  14. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  15. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  16. 2016Optical Flow Estimation using a Spatial Pyramid Networkcited by 421
  17. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  18. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  19. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  20. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  21. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  22. 2020Softmax Splatting for Video Frame Interpolationcited by 276
  23. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 274
  24. 2022FlowFormer: A Transformer Architecture for Optical Flowcited by 273
  25. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  26. 2021Motion Representations for Articulated Animationcited by 253
  27. 2023Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifoldcited by 218
  28. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 215
  29. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 193
  30. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  31. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  32. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  33. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  34. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  35. 2023FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimationcited by 114
  36. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 88
  37. 2023VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimationcited by 82
  38. 2019A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularizationcited by 76

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 41; 2025: 109; 2026: 59.

209 papers.

Sort:

By area:

  1. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  2. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  3. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  4. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  5. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  6. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  7. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  8. 2025Controllable Video Generation: A Surveycited by 93
  9. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
  10. 2025VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Controlcited by 83
  11. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  12. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
38 more
  1. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  2. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 73
  3. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  4. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  5. 2024Rectified Diffusion: Straightness Is Not Your Need in Rectified Flowcited by 59
  6. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Modelscited by 55
  7. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  8. 2024Follow-Your-Click: Open-domain Regional Image Animation via Short Promptscited by 52
  9. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 51
  10. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  11. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 47
  12. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  13. 2025Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidancecited by 45
  14. 2025CamCloneMaster: Enabling Reference-based Camera Control for Video Generationcited by 44
  15. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  16. 2024AnimateLCM: Computation-Efficient Personalized Style Video Generation without Personalized Video Datacited by 42
  17. 2024VBench++: Comprehensive and Versatile Benchmark Suite for Video Generative Modelscited by 42
  18. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 42
  19. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  20. 2025GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields through Efficient Dense 3D Point Trackingcited by 37
  21. 2025FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesiscited by 37
  22. 2024OSV: One Step is Enough for High-Quality Image to Video Generationcited by 36
  23. 2024Generative Video Propagationcited by 36
  24. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 34
  25. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  26. 2025MultiShotMaster: A Controllable Multi-Shot Video Generation Frameworkcited by 33
  27. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  28. 2025ObjectMover: Generative Object Movement with Video Priorcited by 30
  29. 2025MotionPro: A Precise Motion Controller for Image-to-Video Generationcited by 29
  30. 2024Video Diffusion Models: A Surveycited by 28
  31. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  32. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  33. 2025Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representationcited by 22
  34. 2025Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Modelscited by 21
  35. 2025LongVie: Multimodal-Guided Controllable Ultra-Long Video Generationcited by 21
  36. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  37. 2025Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generationcited by 20
  38. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20

50 most cited of 209 All 209 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation
  2. 2024Controllable Longer Image Animation with Diffusion Models
  3. 2025Training-Free Motion-Guided Video Generation with Enhanced Temporal Consistency Using Motion Consistency Loss
  4. 2025Dynamic-I2V: Exploring Image-to-Video Generation Models via Multimodal LLM
  5. 2025LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer
  6. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion
  7. 2024Move-in-2D: 2D-Conditioned Human Motion Generation
  8. 2024Animate Your Motion: Turning Still Images into Dynamic Videos