mlsanityData to 7 Oct 2026

GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions

arXiv 2104.14806 ↗First posted 30 Apr 2021cs.CVcited by 254

Not on the map. Why?

Cited by 95 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Generating videos from text is a challenging task due to its high computational requirements for training and infinite possible answers for evaluation. Existing works typically experiment on simple or small datasets, where the generalization ability is quite limited. In this work, we propose GODIVA, an open-domain text-to-video pretrained model that can generate videos from text in an auto-regressive manner using a three-dimensional sparse attention mechanism. We pretrain our model on Howto100M, a large-scale text-video dataset that contains more than 136 million text-video pairs. Experiments show that GODIVA not only can be fine-tuned on downstream video generation tasks, but also has a good zero-shot capability on unseen texts. We also propose a new metric called Relative Matching (RM) to automatically evaluate the video generation quality. Several challenges are listed and discussed as future work.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 30 Apr 2021.

Builds on

24 of 34 references.

References · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Neural Discrete Representation Learningcited by 4,650
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2017The Kinetics Human Action Video Datasetcited by 1,894
  6. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  7. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  8. 2018Self-Attention Generative Adversarial Networkscited by 809
  9. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  10. 2016Generating Videos with Scene Dynamicscited by 376
  11. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  12. 2018Stochastic Video Generation with a Learned Priorcited by 244
12 more
  1. 2017Stochastic Variational Video Predictioncited by 234
  2. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  3. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  4. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  5. 2017Video Generation From Textcited by 153
  6. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  7. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  8. 2020Latent Video Transformercited by 106
  9. 2018To Create What You Tell: Generating Videos from Captionscited by 84
  10. 2021Predicting Video with VQVAEcited by 61
  11. 2016Neural Paraphrase Generation with Stacked Residual LSTM Networkscited by 52
  12. 2016Sync-DRAW: Automatic Video Generation using Deep Recurrent Attentive Architecturescited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 20; 2023: 74; 2024: 78; 2025: 57; 2026: 18.

254 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  4. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  6. 2023ModelScope Text-to-Video Technical Reportcited by 651
  7. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  8. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  9. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  10. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  11. 2023Learning Interactive Real-World Simulatorscited by 440
  12. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
38 more
  1. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  2. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  3. 2022Transformers are Sample-Efficient World Modelscited by 317
  4. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  5. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  6. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  7. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  8. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  9. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  10. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  11. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  12. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  13. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  14. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  15. 2023A Survey on Video Diffusion Modelscited by 203
  16. 2024BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusioncited by 200
  17. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  18. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  19. 2023NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generationcited by 170
  20. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  21. 2024Behavior Generation with Latent Actionscited by 159
  22. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  23. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  24. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  25. 2023Direct Inversion: Boosting Diffusion-based Editing with 3 Lines of Codecited by 145
  26. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  27. 2023VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planningcited by 138
  28. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  29. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Modelscited by 135
  30. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  31. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 125
  32. 2023FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generationcited by 119
  33. 2024Training-free Camera Control for Video Generationcited by 119
  34. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  35. 2024ViDiT-Q: Efficient and Accurate Quantization of Diffusion Transformers for Image and Video Generationcited by 110
  36. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  37. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  38. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105

50 most cited of 254 All 254 (JSON)