mlsanityData to 7 Oct 2026

AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

arXiv 2307.04725 ↗First posted 10 Jul 2023cs.CV cs.GR cs.LGcited by 1,418

On the map: Diffusion models → Video generation

Abstract (arXiv)

With the advance of text-to-image (T2I) diffusion models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. However, adding motion dynamics to existing high-quality personalized T2Is and enabling them to generate animations remains an open challenge. In this paper, we present AnimateDiff, a practical framework for animating personalized T2I models without requiring model-specific tuning. At the core of our framework is a plug-and-play motion module that can be trained once and seamlessly integrated into any personalized T2Is originating from the same base T2I. Through our proposed training strategy, the motion module effectively learns transferable motion priors from real-world videos. Once trained, the motion module can be inserted into a personalized T2I model to form a personalized animation generator. We further propose MotionLoRA, a lightweight fine-tuning technique for AnimateDiff that enables a pre-trained motion module to adapt to new motion patterns, such as different shot types, at a low training and data collection cost. We evaluate AnimateDiff and MotionLoRA on several public representative personalized T2I models collected from the community. The results demonstrate that our approaches help these models generate temporally smooth animation clips while preserving the visual quality and motion diversity. Codes and pre-trained weights are available at https://github.com/guoyww/AnimateDiff.

Key facts. Cited by 1,418 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 48 of its 59 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

48 of 59 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  2. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  4. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  5. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation22 Mar
  6. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  7. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  8. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  9. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  11. 2022Video Diffusion Models7 Apr

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
25 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  13. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  14. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  15. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  16. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  17. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  18. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  19. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  20. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  21. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  22. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  23. 2023Taming Encoder for Zero Fine-tuning Image Customization with Text-to-Image Diffusion Modelscited by 116
  24. 2021Motion-aware Contrastive Video Representation Learning via Foreground-background Mergingcited by 45
  25. 2022UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidancecited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 81; 2024: 462; 2025: 555; 2026: 320.

1,418 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  5. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  6. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  7. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  8. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  9. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  10. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  11. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  12. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
38 more
  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  3. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  4. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  5. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  6. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  7. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  8. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  9. 2023EvalCrafter: Benchmarking and Evaluating Large Video Generation Modelscited by 332
  10. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  11. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  12. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  13. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  14. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  15. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 271
  16. 2024SDXL-Lightning: Progressive Adversarial Diffusion Distillationcited by 258
  17. 2023Pixel-Aware Stable Diffusion for Realistic Image Super-resolution and Personalized Stylizationcited by 251
  18. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 242
  19. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  20. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  21. 2024EMO: Emote Portrait Alive -- Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditionscited by 233
  22. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 218
  23. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  24. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  25. 2023A Survey on Video Diffusion Modelscited by 203
  26. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  27. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  28. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  29. 2024AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animationcited by 191
  30. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 190
  31. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  32. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  33. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  34. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  35. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  36. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  37. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  38. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172

50 most cited of 1,418 All 1,418 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023LatentMan: Generating Consistent Animated Characters using Image Diffusion Models
  2. 2023Dreamix: Video Diffusion Models are General Video Editors
  3. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising
  4. 2024Towards motion from video diffusion models
  5. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priors
  6. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  7. 2023Single Motion Diffusion
  8. 2023Animate124: Animating One Image to 4D Dynamic Scene