mlsanityData to 7 Oct 2026

Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation

arXiv 2212.11565 ↗First posted 22 Dec 2022cs.CVcited by 958

On the map: Diffusion models → Video generation

Abstract (arXiv)

To replicate the success of text-to-image (T2I) generation, recent works employ large-scale video datasets to train a text-to-video (T2V) generator. Despite their promising results, such paradigm is computationally expensive. In this work, we propose a new T2V generation settingunicodex2014One-Shot Video Tuning, where only one text-video pair is presented. Our model is built on state-of-the-art T2I diffusion models pre-trained on massive image data. We make two key observations: 1) T2I models can generate still images that represent verb terms; 2) extending T2I models to generate multiple images concurrently exhibits surprisingly good content consistency. To further learn continuous motion, we introduce Tune-A-Video, which involves a tailored spatio-temporal attention mechanism and an efficient one-shot tuning strategy. At inference, we employ DDIM inversion to provide structure guidance for sampling. Extensive qualitative and numerical experiments demonstrate the remarkable ability of our method across various applications.

Key facts. Cited by 958 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 51 of its 71 references that are in the map. First posted on arXiv 22 Dec 2022.

Builds on

51 of 71 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022SinFusion: Training Diffusion Models on a Single Image or Video21 Nov
  2. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  4. 2022Video Diffusion Models7 Apr

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
35 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  4. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  7. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  8. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  11. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  12. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  13. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  14. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  15. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  16. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  17. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  18. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  19. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  20. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  21. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  22. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  23. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  24. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  25. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  26. 2017Video Generation From Textcited by 153
  27. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  28. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  29. 2022Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidancecited by 67
  30. 2022MagicMix: Semantic Mixing with Diffusion Modelscited by 63
  31. 2018Imagine This! Scripts to Compositions to Videoscited by 54
  32. 2016Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architecturescited by 44
  33. 2020Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Samplecited by 43
  34. 2017Attentive Semantic Video Generation using Captionscited by 28
  35. 2021Diverse Generation from a Single Video Made Possiblecited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 215; 2024: 327; 2025: 289; 2026: 127.

958 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  3. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  5. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  6. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  7. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  8. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  9. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  10. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  11. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  12. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
38 more
  1. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  2. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  3. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  4. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  5. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  6. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  7. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  8. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  9. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Modelcited by 330
  10. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  11. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  12. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  13. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  14. 2023SeeSR: Towards Semantics-Aware Real-World Image Super-Resolutioncited by 311
  15. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  16. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  17. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  18. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  19. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  20. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  21. 2023MagicDrive: Street View Generation with Diverse 3D Geometry Controlcited by 250
  22. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  23. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  24. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  25. 2023Implicit Diffusion Models for Continuous Super-Resolutioncited by 216
  26. 2024FlowEdit: Inversion-Free Text-Based Editing Using Pre-Trained Flow Modelscited by 214
  27. 2024Diffusion Model-Based Image Editing: A Surveycited by 212
  28. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  29. 2023A Survey on Video Diffusion Modelscited by 203
  30. 2024MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidancecited by 196
  31. 2023FreeU: Free Lunch in Diffusion U-Netcited by 195
  32. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 193
  33. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  34. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  35. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 182
  36. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  37. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  38. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172

50 most cited of 958 All 958 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Video Text Preservation with Synthetic Text-Rich Videos
  2. 2024VideoTetris: Towards Compositional Text-to-Video Generation
  3. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models
  4. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling
  5. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
  6. 2024VideoGuide: Improving Video Diffusion Models without Training Through a Teacher's Guide
  7. 2026UniVid: Pyramid Diffusion Model for High Quality Video Generation
  8. 2024HARIVO: Harnessing Text-to-Image Models for Video Generation