mlsanityData to 7 Oct 2026

SimDA: Simple Diffusion Adapter for Efficient Video Generation

arXiv 2308.09710 ↗First posted 18 Aug 2023cs.CV cs.AIcited by 101

Not on the map. Why?

Cited by 58 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The recent wave of AI-generated content has witnessed the great development and success of Text-to-Image (T2I) technologies. By contrast, Text-to-Video (T2V) still falls short of expectations though attracting increasing interests. Existing works either train from scratch or adapt large T2I model to videos, both of which are computation and resource expensive. In this work, we propose a Simple Diffusion Adapter (SimDA) that fine-tunes only 24M out of 1.1B parameters of a strong T2I model, adapting it to video generation in a parameter-efficient way. In particular, we turn the T2I model for T2V by designing light-weight spatial and temporal adapters for transfer learning. Besides, we change the original spatial attention to the proposed Latent-Shift Attention (LSA) for temporal consistency. With similar model architecture, we further train a video super-resolution model to generate high-definition (1024x1024) videos. In addition to T2V generation in the wild, SimDA could also be utilized in one-shot video editing with only 2 minutes tuning. Doing so, our method could minimize the training effort with extremely few tunable parameters for model adaptation.

Key facts. Cited by 101 papers in the map (data through 7 Oct 2026). Builds on 118 of its 139 references that are in the map. First posted on arXiv 18 Aug 2023.

Builds on

118 of 139 references.

References · 118

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  9. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  10. 2020Denoising Diffusion Implicit Modelscited by 9,655
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  12. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
38 more
  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  13. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  14. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  15. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  16. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  17. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  18. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  19. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  20. 2022Video Diffusion Modelscited by 2,280
  21. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  22. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  23. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  24. 2022Visual Prompt Tuningcited by 1,674
  25. 2021ViViT: A Video Vision Transformercited by 1,610
  26. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  27. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  28. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  29. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  30. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  31. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  32. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  33. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  34. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  35. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  36. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  37. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  38. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897

50 most cited of 118 All 118 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 42; 2025: 27; 2026: 13.

101 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  3. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  4. 2023A Survey on Video Diffusion Modelscited by 203
  5. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  6. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  7. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  8. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 117
  9. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  10. 2024Animate-X: Universal Character Image Animation with Enhanced Motion Representationcited by 86
  11. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  12. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 85
38 more
  1. 2023CCEdit: Creative and Controllable Video Editing via Diffusion Modelscited by 80
  2. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 74
  3. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  4. 2024CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generationcited by 58
  5. 2023MotionEditor: Editing Video Motion via Content-Aware Diffusioncited by 56
  6. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51
  7. 2024On the Content Bias in Fr'echet Video Distancecited by 51
  8. 2023Hierarchical Spatio-temporal Decoupling for Text-to-Video Generationcited by 50
  9. 2024Towards A Better Metric for Text-to-Video Generationcited by 50
  10. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 47
  11. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  12. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  13. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  14. 2025StableAvatar: Infinite-Length Audio-Driven Avatar Video Generationcited by 41
  15. 2024Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slicescited by 40
  16. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generationcited by 39
  17. 2023Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMscited by 35
  18. 2025Efficient Diffusion Models: A Surveycited by 35
  19. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 35
  20. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 33
  21. 2025FlashVideo: Flowing Fidelity to Detail for Efficient High-Resolution Video Generationcited by 32
  22. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  23. 2024TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Modelscited by 30
  24. 2023MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generationcited by 26
  25. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  26. 2025BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformerscited by 24
  27. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  28. 2024EffiVED:Efficient Video Editing via Text-instruction Diffusion Modelscited by 21
  29. 2024Grid Diffusion Models for Text-to-Video Generationcited by 21
  30. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  31. 2024FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audiocited by 20
  32. 2024AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Predictioncited by 20
  33. 2023VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Modelscited by 19
  34. 2023AdaDiff: Adaptive Step Selection for Fast Diffusion Modelscited by 18
  35. 2023Fine-grained Controllable Video Generation via Object Appearance and Contextcited by 18
  36. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  37. 2024REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latentscited by 14
  38. 2024Fast and Memory-Efficient Video Diffusion Using Streamlined Inferencecited by 13

50 most cited of 101 All 101 (JSON)