mlsanityData to 14 Oct 2026

Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation

arXiv 2311.17117 ↗First posted 28 Nov 2023cs.CVcited by 688

On the map: Diffusion models → Video generation

Abstract (arXiv)

Character Animation aims to generating character videos from still images through driving signals. Currently, diffusion models have become the mainstream in visual generation research, owing to their robust generative capabilities. However, challenges persist in the realm of image-to-video, especially in character animation, where temporally maintaining consistency with detailed information from character remains a formidable problem. In this paper, we leverage the power of diffusion models and propose a novel framework tailored for character animation. To preserve consistency of intricate appearance features from reference image, we design ReferenceNet to merge detail features via spatial attention. To ensure controllability and continuity, we introduce an efficient pose guider to direct character's movements and employ an effective temporal modeling approach to ensure smooth inter-frame transitions between video frames. By expanding the training data, our approach can animate arbitrary characters, yielding superior results in character animation compared to other image-to-video methods. Furthermore, we evaluate our method on benchmarks for fashion video and human dance synthesis, achieving state-of-the-art results.

Key facts. Cited by 688 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 57 of its 66 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

57 of 66 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  2. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  4. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  6. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Models24 Mar
  7. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  8. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  9. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  12. 2022Video Diffusion Models7 Apr

Other references · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  8. 2020Denoising Diffusion Implicit Modelscited by 9,781
  9. 2023Segment Anythingcited by 8,234
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
33 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  2. 2017Neural Discrete Representation Learningcited by 4,712
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  5. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,464
  6. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,351
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  8. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  10. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 931
  11. 2020First Order Motion Model for Image Animationcited by 704
  12. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  13. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  14. 2018Everybody Dance Nowcited by 315
  15. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  16. 2023Effective Whole-body Pose Estimation with Two-stages Distillationcited by 311
  17. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  18. 2022EDGE: Editable Dance Generation From Musiccited by 287
  19. 2021Motion Representations for Articulated Animationcited by 255
  20. 2022Thin-Plate Spline Motion Model for Image Animationcited by 210
  21. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 208
  22. 2020Searching Central Difference Convolutional Networks for Face Anti-Spoofingcited by 173
  23. 2019Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesiscited by 161
  24. 2023TryOnDiffusion: A Tale of Two UNetscited by 141
  25. 2019DwNet: Dense warp-based network for pose-guided human video generationcited by 115
  26. 2022Person Image Synthesis via Denoising Diffusion Modelcited by 104
  27. 2015face anti-spoofing based on color texture analysiscited by 82
  28. 2020Deep Spatial Gradient and Temporal Depth Learning for Face Anti-spoofingcited by 70
  29. 2022Exploring Dual-task Correlation for Pose Guided Person Image Generationcited by 68
  30. 2023Learning Locally Editable Virtual Humanscited by 54
  31. 2020Deep Spatial Transformation for Pose-Guided Person Image Generation and Animationcited by 43
  32. 2023LEO: Generative Latent Image Animator for Human Video Synthesiscited by 40
  33. 2023Bidirectionally Deformable Motion Modulation For Video-based Human Pose Transfercited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 206; 2025: 327; 2026: 151.

688 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 477
  3. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  4. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 272
  5. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 247
  6. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 235
  7. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  8. 2024LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Controlcited by 206
  9. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 198
  10. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  11. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 194
  12. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
38 more
  1. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  2. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 174
  3. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  4. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 159
  5. 2024OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-oncited by 151
  6. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  7. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 129
  8. 2024Training-Free Consistent Text-to-Image Generationcited by 125
  9. 2024ID-Animator: Zero-Shot Identity-Preserving Human Video Generationcited by 124
  10. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 120
  11. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  12. 2024Improving Diffusion Models for Authentic Virtual Try-on in the Wildcited by 109
  13. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 102
  14. 2024Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion Dependencycited by 102
  15. 2025Controllable Video Generation: A Surveycited by 98
  16. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 97
  17. 2023VideoLCM: Video Latent Consistency Modelcited by 92
  18. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 92
  19. 2024X-Portrait: Expressive Portrait Animation with Hierarchical Motion Attentioncited by 91
  20. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 90
  21. 2024Animate-X: Universal Character Image Animation with Enhanced Motion Representationcited by 88
  22. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 88
  23. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  24. 2025Wan-Animate: Unified Character Animation and Replacement with Holistic Replicationcited by 80
  25. 2024OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generationcited by 77
  26. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 77
  27. 2024Ca2-VDM: Efficient Autoregressive Video Diffusion Model with Causal Generation and Cache Sharingcited by 74
  28. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 74
  29. 2024GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warpingcited by 73
  30. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  31. 2024EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animationcited by 72
  32. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70
  33. 2024Progressive Autoregressive Video Diffusion Modelscited by 68
  34. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  35. 2024Deepfake Generation and Detection: A Benchmark and Surveycited by 67
  36. 2024AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encodingcited by 67
  37. 2024Face Adapter for Pre-Trained Diffusion Models with Fine-Grained ID and Attribute Controlcited by 67
  38. 2024Zero-shot Image Editing with Reference Imitationcited by 65

50 most cited of 688 All 688 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MultiAnimate: Pose-Guided Image Animation Made Extensible
  2. 2023AnimateZero: Video Diffusion Models are Zero-Shot Image Animators
  3. 2023DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion
  4. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
  5. 2025AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Models
  6. 2024Towards motion from video diffusion models
  7. 2025Generating Detailed Character Motion from Blocking Poses
  8. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising