mlsanityData to 14 Oct 2026

MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model

arXiv 2405.20222 ↗First posted 30 May 2024cs.CV cs.AIcited by 89

On the map: Diffusion models → Video generation

Abstract (arXiv)

We present MOFA-Video, an advanced controllable image animation method that generates video from the given image using various additional controllable signals (such as human landmarks reference, manual trajectories, and another even provided video) or their combinations. This is different from previous methods which only can work on a specific motion domain or show weak control abilities with diffusion prior. To achieve our goal, we design several domain-aware motion field adapters (ie, MOFA-Adapters) to control the generated motions in the video generation pipeline. For MOFA-Adapters, we consider the temporal motion consistency of the video and generate the dense motion flow from the given sparse control conditions first, and then, the multi-scale features of the given image are wrapped as a guided feature for stable video diffusion generation. We naively train two motion adapters for the manual trajectories and the human landmarks individually since they both contain sparse information about the control. After training, the MOFA-Adapters in different domains can also work together for more controllable video generation. Project Page: https://myniuuu.github.io/MOFAVideo/

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 39 of its 48 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

39 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models25 Mar
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  3. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors18 Oct
  5. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  6. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  7. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Models24 Mar

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  9. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  10. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 382
  11. 2019SinGAN: Learning a Generative Model from a Single Natural Imagecited by 348
  12. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 311
20 more
  1. 2022Unifying Flow, Stereo and Depth Estimationcited by 275
  2. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  3. 2021Motion Representations for Articulated Animationcited by 255
  4. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 211
  5. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  6. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 175
  7. 2022StyleHEAT: One-Shot High-Resolution Editable Talking Face Generation via Pre-trained StyleGANcited by 169
  8. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  9. 2023Generative Image Dynamicscited by 94
  10. 2017Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networkscited by 89
  11. 2018Photo Wake-Up: 3D Character Animation from a Single Photocited by 60
  12. 2018Flow-Grounded Spatial-Temporal Video Prediction from Still Imagescited by 55
  13. 2021Stochastic Image-to-Video Synthesis using cINNscited by 49
  14. 2021Controllable Animation of Fluid Elements in Still Imagescited by 45
  15. 2021iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesiscited by 42
  16. 2021Understanding Object Dynamics for Interactive Image-to-Video Synthesiscited by 38
  17. 2023Text-Guided Synthesis of Eulerian Cinemagraphscited by 22
  18. 2019Self-Supervised Learning via Conditional Motion Propagationcited by 20
  19. 2023Blowing in the Wind: CycleNet for Human Cinemagraphs from Still Imagescited by 14
  20. 2023Automatic Animation of Hair Blowing in Still Portrait Photoscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 40; 2026: 28.

89 papers.

Sort:

By area:

  1. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 403
  2. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 185
  3. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  4. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 105
  5. 2025Cameras as Relative Positional Encodingcited by 105
  6. 2025Controllable Video Generation: A Surveycited by 98
  7. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  8. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 76
  9. 2025PhysGen3D: Crafting a Miniature Interactive World from a Single Imagecited by 66
  10. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 53
  11. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 52
  12. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
38 more
  1. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 45
  2. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 43
  3. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 42
  4. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  5. 2025GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video Diffusion Priorscited by 38
  6. 2024StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videoscited by 37
  7. 2024PhysFlow: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulationcited by 36
  8. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 35
  9. 2024DisPose: Disentangling Pose Guidance for Controllable Human Image Animationcited by 35
  10. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  11. 2025MotionPro: A Precise Motion Controller for Image-to-Video Generationcited by 30
  12. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  13. 2024StableV2V: Stablizing Shape Consistency in Video-to-Video Editingcited by 26
  14. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  15. 2025Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representationcited by 23
  16. 2025Generative Video Motion Editing with 3D Point Trackscited by 23
  17. 2026RealWonder: Real-Time Physical Action-Conditioned Video Generationcited by 22
  18. 2024I2VControl: Disentangled and Unified Video Motion Synthesis Controlcited by 20
  19. 2025GeoDrive: 3D Geometry-Informed Driving World Model with Precise Action Controlcited by 19
  20. 2024MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generationcited by 16
  21. 2024SurgSora: Object-Aware Diffusion Model for Controllable Surgical Video Generationcited by 15
  22. 2025MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-oncited by 15
  23. 2025FaceShot: Bring Any Character into Lifecited by 13
  24. 2024Motion Modes: What Could Happen Next?cited by 12
  25. 2024Video Diffusion Transformers are In-Context Learnerscited by 12
  26. 2025TransPixeler: Advancing Text-to-Video Generation with Transparencycited by 12
  27. 20253D-Fixup: Advancing Photo Editing with 3D Priorscited by 12
  28. 2025MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agentcited by 11
  29. 2025AnyI2V: Animating Any Conditional Image with Motion Controlcited by 11
  30. 2025AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Modelscited by 10
  31. 2025T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitratescited by 10
  32. 2026Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goalscited by 9
  33. 2026PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generationcited by 9
  34. 2025Consistent Video Editing as Flow-Driven Image-to-Video Generationcited by 8
  35. 2024ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Waycited by 7
  36. 2024MikuDance: Animating Character Art with Mixed Motion Dynamicscited by 7
  37. 2025Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometrycited by 7
  38. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Modelscited by 7

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Controllable Longer Image Animation with Diffusion Models
  2. 2023MoVideo: Motion-Aware Video Generation with Diffusion Models
  3. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising
  4. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis
  5. 2024Video Diffusion Models are Training-free Motion Interpreter and Controller
  6. 2026Moaw: Unleashing Motion Awareness for Video Diffusion Models
  7. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generation
  8. 2026MotiMotion: Motion-Controlled Video Generation with Visual Reasoning