mlsanityData to 7 Oct 2026

Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation

arXiv 2310.05737 ↗First posted 9 Oct 2023cs.CV cs.AI cs.MMcited by 633

On the map: Diffusion models → Video generation

Abstract (arXiv)

While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation. To effectively use LLMs for visual generation, one crucial component is the visual tokenizer that maps pixel-space inputs to discrete tokens appropriate for LLM learning. In this paper, we introduce MAGVIT-v2, a video tokenizer designed to generate concise and expressive tokens for both videos and images using a common token vocabulary. Equipped with this new tokenizer, we show that LLMs outperform diffusion models on standard image and video generation benchmarks including ImageNet and Kinetics. In addition, we demonstrate that our tokenizer surpasses the previously top-performing video tokenizer on two more tasks: (1) video compression comparable to the next-generation video codec (VCC) according to human evaluations, and (2) learning effective representations for action recognition tasks.

Key facts. Cited by 633 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 71 of its 82 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

71 of 82 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  3. 2022Scalable Adaptive Computation for Iterative Generation22 Dec
  4. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  5. 2022Video Diffusion Models7 Apr

Other references · 66

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2023GPT-4 Technical Reportcited by 18,132
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
38 more
  1. 2022Scalable Diffusion Models with Transformerscited by 5,931
  2. 2022Classifier-Free Diffusion Guidancecited by 5,717
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  10. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  13. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  14. 2017The Kinetics Human Action Video Datasetcited by 1,894
  15. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  16. 2021ViViT: A Video Vision Transformercited by 1,610
  17. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  18. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  19. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  20. 2023PaLM 2 Technical Reportcited by 1,154
  21. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  22. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  23. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  24. 2023StarCoder: may the source be with you!cited by 790
  25. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  26. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  27. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  28. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  29. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  30. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  31. 2022StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasetscited by 489
  32. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  33. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 413
  34. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  35. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  36. 2022MAGVIT: Masked Generative Video Transformercited by 341
  37. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  38. 2018A Short Note about Kinetics-600cited by 331

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 176; 2025: 287; 2026: 161.

633 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,536
  2. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  5. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  6. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  7. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  8. 2024Autoregressive Image Generation without Vector Quantizationcited by 644
  9. 2024Open-Sora: Democratizing Efficient Video Production for Allcited by 637
  10. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  11. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  12. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
38 more
  1. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  2. 2025Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Modelscited by 376
  3. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  4. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  5. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  6. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  7. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 276
  8. 2025Diffusion Transformers with Representation Autoencoderscited by 275
  9. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 247
  10. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  11. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  12. 2025REPA-E: Unlocking VAE for End-to-End Tuning with Latent Diffusion Transformerscited by 212
  13. 2024Deep Compression Autoencoder for Efficient High-Resolution Diffusion Modelscited by 209
  14. 2024How Far is Video Generation from World Model: A Physical Law Perspectivecited by 204
  15. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  16. 2024EchoMimic: Lifelike Audio-Driven Portrait Animations through Editable Landmark Conditionscited by 196
  17. 2024T2V-CompBench: A Comprehensive Benchmark for Compositional Text-to-video Generationcited by 177
  18. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  19. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  20. 2025History-Guided Video Diffusioncited by 165
  21. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 163
  22. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  23. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 159
  24. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153
  25. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  26. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  27. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 127
  28. 2024Randomized Autoregressive Visual Generationcited by 125
  29. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 124
  30. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  31. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 117
  32. 2024Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesiscited by 116
  33. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  34. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  35. 2024Image and Video Tokenization with Binary Spherical Quantizationcited by 110
  36. 2025Long Context Tuning for Video Generationcited by 110
  37. 2024Rolling Diffusion Modelscited by 102
  38. 2024Loong: Generating Minute-level Long Videos with Autoregressive Language Modelscited by 99

50 most cited of 633 All 633 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
  2. 2024Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation
  3. 2026Enhancing Visual Token Representations for Video Large Language Models via Training-Free Spatial-Temporal Pooling and Gridding
  4. 2023Planting a SEED of Vision in Large Language Model
  5. 2026Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline
  6. 2025TokensGen: Harnessing Condensed Tokens for Long Video Generation
  7. 2024TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models
  8. 2025The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation