mlsanityData to 7 Oct 2026

CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

arXiv 2205.15868 ↗First posted 29 May 2022cs.CV cs.CL cs.LGcited by 973

Not on the map. Why?

Cited by 391 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large-scale pretrained transformers have created milestones in text (GPT-3) and text-to-image (DALL-E and CogView) generation. Its application to video generation is still facing many challenges: The potential huge computation cost makes the training from scratch unaffordable; The scarcity and weak relevance of text-video datasets hinder the model understanding complex movement semantics. In this work, we present 9B-parameter transformer CogVideo, trained by inheriting a pretrained text-to-image model, CogView2. We also propose multi-frame-rate hierarchical training strategy to better align text and video clips. As (probably) the first open-source large-scale pretrained text-to-video model, CogVideo outperforms all publicly available models at a large margin in machine and human evaluations.

Key facts. Cited by 973 papers in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 29 May 2022.

Builds on

35 of 45 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2022Video Diffusion Modelscited by 2,280
23 more
  1. 2020Extracting Training Data from Large Language Modelscited by 2,144
  2. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 852
  5. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  6. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  7. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  8. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  9. 2016Generating Videos with Scene Dynamicscited by 376
  10. 2018A Short Note about Kinetics-600cited by 331
  11. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  12. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 256
  13. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  14. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  15. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  16. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  17. 2019Scaling Autoregressive Video Modelscited by 189
  18. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  19. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  20. 2018Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GANcited by 120
  21. 2020Latent Video Transformercited by 106
  22. 2020Transformation-based Adversarial Video Prediction on Large-Scale Datacited by 66
  23. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 143; 2024: 222; 2025: 366; 2026: 225.

973 papers.

Sort:

By area:

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2024Genie: Generative Interactive Environmentscited by 706
  10. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  11. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
38 more
  1. 2023ModelScope Text-to-Video Technical Reportcited by 651
  2. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  5. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  6. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  7. 2024Latte: Latent Diffusion Transformer for Video Generationcited by 536
  8. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  9. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  10. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  13. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  14. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  15. 2023MusicLM: Generating Music From Textcited by 442
  16. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  17. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  18. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  19. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 385
  20. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  21. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  22. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  23. 2022MAGVIT: Masked Generative Video Transformercited by 341
  24. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  25. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 324
  26. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  27. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  28. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  29. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  30. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  31. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  32. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  33. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  34. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  35. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  36. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  37. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  38. 2022CelebV-HQ: A Large-Scale Video Facial Attributes Datasetcited by 246

50 most cited of 973 All 973 (JSON)