mlsanityData to 7 Oct 2026

Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators

arXiv 2303.13439 ↗First posted 23 Mar 2023cs.CVcited by 666

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent text-to-video generation approaches rely on computationally heavy training and require large-scale video datasets. In this paper, we introduce a new task of zero-shot text-to-video generation and propose a low-cost approach (without any training or optimization) by leveraging the power of existing text-to-image synthesis methods (e.g., Stable Diffusion), making them suitable for the video domain. Our key modifications include (i) enriching the latent codes of the generated frames with motion dynamics to keep the global scene and the background time consistent; and (ii) reprogramming frame-level self-attention using a new cross-frame attention of each frame on the first frame, to preserve the context, appearance, and identity of the foreground object. Experiments show that this leads to low overhead, yet high-quality and remarkably consistent video generation. Moreover, our approach is not limited to text-to-video synthesis but is also applicable to other tasks such as conditional and content-specialized video generation, and Video Instruct-Pix2Pix, i.e., instruction-guided video editing. As experiments show, our method performs comparably or sometimes better than recent approaches, despite not being trained on additional video data. Our code will be open sourced at: https://github.com/Picsart-AI-Research/Text2Video-Zero .

Key facts. Cited by 666 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 45 of its 52 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

45 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  2. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  3. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  5. 2022Video Diffusion Models7 Apr

Other references · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2020Denoising Diffusion Implicit Modelscited by 9,655
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
28 more
  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  7. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  13. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  14. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  15. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  16. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  17. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  18. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  19. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  20. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  21. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  22. 2021More Control for Free! Image Synthesis with Semantic Diffusion Guidancecited by 211
  23. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  24. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  25. 2019Salient Object Detection in the Deep Learning Era: An In-Depth Surveycited by 163
  26. 2015Generating Images from Captions with Attentioncited by 135
  27. 2016Learning What and Where to Drawcited by 105
  28. 2023Shape-aware Text-driven Layered Video Editingcited by 54

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 139; 2024: 261; 2025: 181; 2026: 85.

666 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  4. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  6. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  7. 2025VACE: All-in-One Video Creation and Editingcited by 467
  8. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  9. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  10. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  11. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  12. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 332
38 more
  1. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  2. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  3. 2023Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Modelscited by 297
  4. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  5. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  6. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
  7. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  8. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
  9. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 242
  10. 2024StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generationcited by 240
  11. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  12. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  13. 2025Video models are zero-shot learners and reasonerscited by 208
  14. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  15. 2023A Survey on Video Diffusion Modelscited by 203
  16. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  17. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  18. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  19. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  20. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  21. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  22. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Modelscited by 169
  23. 2024Delta-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformerscited by 169
  24. 2023FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editingcited by 165
  25. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  26. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  27. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  28. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  29. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  30. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  31. 2023LLM4Drive: A Survey of Large Language Models for Autonomous Drivingcited by 136
  32. 2023Gen-L-Video: Multi-Text to Long Video Generation via Temporal Co-Denoisingcited by 135
  33. 2024Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Contentcited by 132
  34. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  35. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  36. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  37. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  38. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124

50 most cited of 666 All 666 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026UniVid: Pyramid Diffusion Model for High Quality Video Generation
  2. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  3. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models
  4. 2024I4VGen: Image as Free Stepping Stone for Text-to-Video Generation
  5. 2024VideoTetris: Towards Compositional Text-to-Video Generation
  6. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  7. 2024Grid Diffusion Models for Text-to-Video Generation
  8. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models