mlsanityData to 7 Oct 2026

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

arXiv 2311.16498 ↗First posted 27 Nov 2023cs.CV cs.GRcited by 330

On the map: Diffusion models → Video generation

Abstract (arXiv)

This paper studies the human image animation task, which aims to generate a video of a certain reference identity following a particular motion sequence. Existing animation works typically employ the frame-warping technique to animate the reference image towards the target motion. Despite achieving reasonable results, these approaches face challenges in maintaining temporal consistency throughout the animation due to the lack of temporal modeling and poor preservation of reference identity. In this work, we introduce MagicAnimate, a diffusion-based framework that aims at enhancing temporal consistency, preserving reference image faithfully, and improving animation fidelity. To achieve this, we first develop a video diffusion model to encode temporal information. Second, to maintain the appearance coherence across frames, we introduce a novel appearance encoder to retain the intricate details of the reference image. Leveraging these two innovations, we further employ a simple video fusion technique to encourage smooth transitions for long video animation. Empirical results demonstrate the superiority of our method over baseline approaches on two benchmarks. Notably, our approach outperforms the strongest baseline by over 38% in terms of video fidelity on the challenging TikTok dancing dataset. Code and model will be made available.

Key facts. Cited by 330 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 45 of its 60 references that are in the map. First posted on arXiv 27 Nov 2023.

Builds on

45 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  2. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoising29 May
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  4. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  9. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  11. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  12. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
29 more
  1. 2020Face2Face: Real-time Face Capture and Reenactment of RGB Videoscited by 631
  2. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 573
  3. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  4. 2020One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencingcited by 438
  5. 2019FSGAN: Subject Agnostic Face Swapping and Reenactmentcited by 319
  6. 2018Everybody Dance Nowcited by 313
  7. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  8. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 307
  9. 2019Few-Shot Adversarial Learning of Realistic Neural Talking Head Modelscited by 284
  10. 2021Motion Representations for Articulated Animationcited by 253
  11. 2022Thin-Plate Spline Motion Model for Image Animationcited by 209
  12. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 206
  13. 2021PIRenderer: Controllable Portrait Image Generation via Semantic Neural Renderingcited by 202
  14. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  15. 2022Latent Image Animator: Learning to Animate Images via Latent Space Navigationcited by 172
  16. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusioncited by 167
  17. 2020Fast Bi-layer Neural Synthesis of One-Shot Realistic Head Avatarscited by 116
  18. 2022EVA3D: Compositional 3D Human Generation from 2D Image Collectionscited by 111
  19. 2021Modeling Clothing as a Separate Layer for an Animatable Human Avatarcited by 66
  20. 2022AvatarGen: A 3D Generative Model for Animatable Human Avatarscited by 59
  21. 20193D Guided Fine-Grained Face Manipulationcited by 45
  22. 2022Implicit Warping for Animation with Image Setscited by 45
  23. 2023LEO: Generative Latent Image Animator for Human Video Synthesiscited by 40
  24. 2023OmniAvatar: Geometry-Guided Controllable 3D Head Synthesiscited by 34
  25. 2019Make a Face: Towards Arbitrary High Fidelity Face Manipulationcited by 29
  26. 2022PV3D: A 3D Generative Model for Portrait Video Generationcited by 23
  27. 2023Unsupervised Volumetric Animationcited by 18
  28. 2023MagicAvatar: Multimodal Avatar Generation and Animationcited by 17
  29. 2023XAGen: 3D Expressive Human Avatars Generationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 111; 2025: 146; 2026: 70.

330 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  3. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271
  4. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  5. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 202
  6. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  7. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  8. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  9. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  10. 2024Training-Free Consistent Text-to-Image Generationcited by 123
  11. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  12. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
38 more
  1. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  2. 2024IRASim: A Fine-Grained World Model for Robot Manipulationcited by 93
  3. 2024Expressive Whole-Body 3D Gaussian Avatarcited by 89
  4. 2024X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attentioncited by 88
  5. 2024Animate-X: Universal Character Image Animation with Enhanced Motion Representationcited by 86
  6. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  7. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 85
  8. 2025Wan-Animate: Unified Character Animation and Replacement with Holistic Replicationcited by 76
  9. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
  10. 2024GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warpingcited by 72
  11. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  12. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70
  13. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 69
  14. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  15. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 66
  16. 2024Zero-shot Image Editing with Reference Imitationcited by 64
  17. 2024HumanVid: Demystifying Training Data for Camera-controllable Human Image Animationcited by 63
  18. 2024MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modelingcited by 63
  19. 2025Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordancecited by 60
  20. 2024IDOL: Instant Photorealistic 3D Human Creation from a Single Imagecited by 55
  21. 2024HumanSplat: Generalizable Single-Image Human Gaussian Splatting with Structure Priorscited by 54
  22. 2024Fr'echet Video Motion Distance: A Metric for Evaluating Motion Consistency in Videoscited by 54
  23. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  24. 2024Coarse-to-Fine Latent Diffusion for Pose-Guided Person Image Synthesiscited by 48
  25. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  26. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  27. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  28. 2024Wear-Any-Way: Manipulable Virtual Try-on via Sparse Correspondence Alignmentcited by 42
  29. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 41
  30. 2024SF-V: Single Forward Video Generation Modelcited by 40
  31. 2025UniAnimate-DiT: Human Image Animation with Large-Scale Video Diffusion Transformercited by 40
  32. 2024TrackGo: A Flexible and Efficient Method for Controllable Video Generationcited by 39
  33. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  34. 2024Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videoscited by 38
  35. 2024ViViD: Video Virtual Try-on using Diffusion Modelscited by 38
  36. 2024CyberHost: Taming Audio-driven Avatar Diffusion Model with Region Codebook Attentioncited by 38
  37. 2025HumanDiT: Pose-Guided Diffusion Transformer for Long-form Human Motion Video Generationcited by 38
  38. 2024RealisDance: Equip controllable character animation with realistic handscited by 37

50 most cited of 330 All 330 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation
  2. 2023Bidirectional Temporal Diffusion Model for Temporally Consistent Human Animation
  3. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising
  4. 2024Towards motion from video diffusion models
  5. 2025High-Fidelity and Long-Duration Human Image Animation with Diffusion Transformer
  6. 2024Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation
  7. 2024LoopAnimate: Loopable Salient Object Animation
  8. 2023Dreamix: Video Diffusion Models are General Video Editors