mlsanityData to 7 Oct 2026

Make-A-Video: Text-to-Video Generation without Text-Video Data

arXiv 2209.14792 ↗First posted 29 Sep 2022cs.CV cs.AI cs.LGcited by 1,725

Not on the map. Why?

Cited by 825 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose Make-A-Video -- an approach for directly translating the tremendous recent progress in Text-to-Image (T2I) generation to Text-to-Video (T2V). Our intuition is simple: learn what the world looks like and how it is described from paired text-image data, and learn how the world moves from unsupervised video footage. Make-A-Video has three advantages: (1) it accelerates training of the T2V model (it does not need to learn visual and multimodal representations from scratch), (2) it does not require paired text-video data, and (3) the generated videos inherit the vastness (diversity in aesthetic, fantastical depictions, etc.) of today's image generation models. We design a simple yet effective way to build on T2I models with novel and effective spatial-temporal modules. First, we decompose the full temporal U-Net and attention tensors and approximate them in space and time. Second, we design a spatial temporal pipeline to generate high resolution and frame rate videos with a video decoder, interpolation model and two super resolution models that can enable various applications besides T2V. In all aspects, spatial and temporal resolution, faithfulness to text, and quality, Make-A-Video sets the new state-of-the-art in text-to-video generation, as determined by both qualitative and quantitative measures.

Key facts. Cited by 1,725 papers in the map (data through 7 Oct 2026). Builds on 42 of its 51 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

42 of 51 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022Generative Adversarial Networkscited by 5,603
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  12. 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
30 more
  1. 2022Video Diffusion Modelscited by 2,280
  2. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  5. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  6. 2016Generative Adversarial Text to Image Synthesiscited by 735
  7. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  8. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  9. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  10. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  11. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  12. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  13. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  14. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  15. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  16. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  17. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  18. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  19. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  20. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  21. 2017Video Generation From Textcited by 153
  22. 2022FILM: Frame Interpolation for Large Motioncited by 149
  23. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  24. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  25. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  26. 2018Imagine This! Scripts to Compositions to Videoscited by 54
  27. 2016Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architecturescited by 44
  28. 2017Attentive Semantic Video Generation using Captionscited by 28
  29. 2022MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENerationcited by 20
  30. 20183D Depthwise Convolution: Reducing Model Parameters in 3D Vision Taskscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 34; 2023: 391; 2024: 537; 2025: 481; 2026: 282.

1,725 papers.

Sort:

By area:

  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  4. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  6. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  7. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  9. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  10. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  11. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  12. 2023Training Diffusion Models with Reinforcement Learningcited by 939
38 more
  1. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  2. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  3. 2024Genie: Generative Interactive Environmentscited by 706
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  5. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  6. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  7. 2023ModelScope Text-to-Video Technical Reportcited by 651
  8. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  9. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  11. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  12. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  13. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  14. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  15. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  16. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  17. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  18. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  19. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  20. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 465
  21. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  22. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  23. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  24. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  25. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  26. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  27. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  28. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  29. 2023Learning Interactive Real-World Simulatorscited by 440
  30. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  31. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  32. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  33. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  34. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  35. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  36. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  37. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  38. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349

50 most cited of 1,725 All 1,725 (JSON)