mlsanityData to 7 Oct 2026

CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

arXiv 2408.06072 ↗First posted 12 Aug 2024cs.CVcited by 2,198

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

We present CogVideoX, a large-scale text-to-video generation model based on diffusion transformer, which can generate 10-second continuous videos aligned with text prompt, with a frame rate of 16 fps and resolution of 768 * 1360 pixels. Previous video generation models often had limited movement and short durations, and is difficult to generate videos with coherent narratives based on text. We propose several designs to address these issues. First, we propose a 3D Variational Autoencoder (VAE) to compress videos along both spatial and temporal dimensions, to improve both compression rate and video fidelity. Second, to improve the text-video alignment, we propose an expert transformer with the expert adaptive LayerNorm to facilitate the deep fusion between the two modalities. Third, by employing a progressive training and multi-resolution frame pack technique, CogVideoX is adept at producing coherent, long-duration, different shape videos characterized by significant motions. In addition, we develop an effective text-video data processing pipeline that includes various data preprocessing strategies and a video captioning method, greatly contributing to the generation quality and semantic alignment. Results show that CogVideoX demonstrates state-of-the-art performance across both multiple machine metrics and human evaluations. The model weight of both 3D Causal VAE, Video caption model and CogVideoX are publicly available at https://github.com/THUDM/CogVideo.

Key facts. Cited by 2,198 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 43 of its 50 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

43 of 50 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  2. 2022Scalable Diffusion Models with Transformers19 Dec
  3. 2022MAGVIT: Masked Generative Video Transformer10 Dec
  4. 2021VideoGPT: Video Generation using VQ-VAE and Transformers20 Apr
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
26 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  6. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  11. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  12. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  13. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  14. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  15. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  16. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  17. 2023Common Diffusion Noise Schedules and Sample Steps are Flawedcited by 308
  18. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  19. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  20. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  21. 2022Generating Videos with Dynamics-aware Implicit Generative Adversarial Networkscited by 186
  22. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  23. 2024T2V-Turbo: Breaking the Quality Bottleneck of Video Consistency Model with Mixed Reward Feedbackcited by 84
  24. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  25. 2024CogView3: Finer and Faster Text-to-Image Generation via Relay Diffusioncited by 72
  26. 2024Evaluation of Text-to-Video Generation Models: A Dynamics Perspectivecited by 66

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 180; 2025: 1027; 2026: 991.

2,198 papers.

Sort:

By area:

  1. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  5. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  6. 2024LTX-Video: Realtime Video Latent Diffusioncited by 549
  7. 2025VACE: All-in-One Video Creation and Editingcited by 467
  8. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
  9. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  10. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  11. 2025VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulnesscited by 289
  12. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
38 more
  1. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  2. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  3. 2026Cosmos Policy: Fine-Tuning Video Models for Visuomotor Control and Planningcited by 274
  4. 2025Improving Video Generation with Human Feedbackcited by 267
  5. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 247
  6. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 245
  7. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  8. 2025LongLive: Real-time Interactive Long Video Generationcited by 237
  9. 2025SkyReels-V2: Infinite-length Film Generative Modelcited by 218
  10. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  11. 2024Taming Rectified Flow for Inversion and Editingcited by 200
  12. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  13. 2025World Simulation with Video Foundation Models for Physical AIcited by 183
  14. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  15. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  16. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  17. 2024Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generationcited by 167
  18. 2025History-Guided Video Diffusioncited by 165
  19. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  20. 2024Understanding World or Predicting Future? A Comprehensive Survey of World Modelscited by 161
  21. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  22. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  23. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  24. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  25. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  26. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  27. 2024VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generationcited by 148
  28. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  29. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  30. 2025VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generationcited by 141
  31. 2024Real-Time Video Generation with Pyramid Attention Broadcastcited by 140
  32. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  33. 2025Genie Envisioner: A Unified World Foundation Platform for Robotic Manipulationcited by 129
  34. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  35. 2025Phantom: Subject-consistent video generation via cross-modal alignmentcited by 126
  36. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  37. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  38. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124

50 most cited of 2,198 All 2,198 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations
  2. 2024Grid Diffusion Models for Text-to-Video Generation
  3. 2023MEVG: Multi-event Video Generation with Text-to-Video Models
  4. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  5. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models
  6. 2024MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion
  7. 2024VideoTetris: Towards Compositional Text-to-Video Generation
  8. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate