mlsanityData to 7 Oct 2026

Vector Quantized Diffusion Model for Text-to-Image Synthesis

arXiv 2111.14822 ↗First posted 29 Nov 2021cs.CV cs.LGcited by 773

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

We present the vector quantized diffusion (VQ-Diffusion) model for text-to-image generation. This method is based on a vector quantized variational autoencoder (VQ-VAE) whose latent space is modeled by a conditional variant of the recently developed Denoising Diffusion Probabilistic Model (DDPM). We find that this latent-space method is well-suited for text-to-image generation tasks because it not only eliminates the unidirectional bias with existing methods but also allows us to incorporate a mask-and-replace diffusion strategy to avoid the accumulation of errors, which is a serious problem with existing methods. Our experiments show that the VQ-Diffusion produces significantly better text-to-image generation results when compared with conventional autoregressive (AR) models with similar numbers of parameters. Compared with previous GAN-based text-to-image methods, our VQ-Diffusion can handle more complex scenes and improve the synthesized image quality by a large margin. Finally, we show that the image generation computation in our method can be made highly efficient by reparameterization. With traditional AR methods, the text-to-image generation time increases linearly with the output image resolution and hence is quite time consuming even for normal size images. The VQ-Diffusion allows us to achieve a better trade-off between quality and speed. Our experiments indicate that the VQ-Diffusion model with the reparameterization is fifteen times faster than traditional AR methods while achieving a better image quality.

Key facts. Cited by 773 papers in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 53 of its 74 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

53 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  2. 2021Cascaded Diffusion Models for High Fidelity Image Generation30 May
  3. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  4. 2021Improved Denoising Diffusion Probabilistic Models18 Feb
  5. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  8. 2016Layer Normalizationcited by 5,579
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  11. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
36 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  4. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  5. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2016Pixel Recurrent Neural Networkscited by 814
  8. 2018Image Transformercited by 770
  9. 2016Generative Adversarial Text to Image Synthesiscited by 735
  10. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  11. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  12. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  13. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  14. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  15. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  16. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  17. 2019Controllable Text-to-Image Generationcited by 187
  18. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  19. 2019Generalization in Generation: A closer look at Exposure Biascited by 123
  20. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  21. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  22. 2018Inferring Semantic Layout for Hierarchical Text-to-Image Synthesiscited by 101
  23. 2021M6: A Chinese Multimodal Pretrainercited by 91
  24. 2018Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Networkcited by 64
  25. 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
  26. 2019Semantics Disentangling for Text-to-Image Generationcited by 61
  27. 2019Mask-Guided Portrait Editing with Conditional GANscited by 61
  28. 2020GIQA: Generated Image Quality Assessmentcited by 51
  29. 2021DAE-GAN: Dynamic Aspect-aware GAN for Text-to-Image Synthesiscited by 45
  30. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41
  31. 2017TAC-GAN - Text Conditioned Auxiliary Classifier Generative Adversarial Networkcited by 21
  32. 2018ChatPainter: Improving Text to Image Generation using Dialoguecited by 18
  33. 2019Dual Adversarial Inference for Text-to-Image Synthesiscited by 17
  34. 2018Adversarial Learning of Semantic Relevance in Text to Image Synthesiscited by 14
  35. 2020Efficient Neural Architecture for Text-to-Image Synthesiscited by 11
  36. 2020PriorGAN: Real Data Prior for Generative Adversarial Netscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 75; 2023: 241; 2024: 226; 2025: 146; 2026: 83.

773 papers.

Sort:

By area:

  1. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  2. 2022Scalable Diffusion Models with Transformerscited by 5,931
  3. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  4. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  5. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  9. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,107
  10. 2022Diffusion Models in Vision: A Surveycited by 1,008
  11. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  12. 2022DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Modelscited by 924
38 more
  1. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  2. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  3. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  4. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  5. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 666
  6. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  7. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  8. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  9. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  10. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  11. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  12. 2023Exploiting Diffusion Prior for Real-World Image Super-Resolutioncited by 486
  13. 2023UniPC: A Unified Predictor-Corrector Framework for Fast Sampling of Diffusion Modelscited by 441
  14. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  15. 2022Blended Latent Diffusioncited by 430
  16. 2023LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPScited by 420
  17. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  18. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 386
  19. 2023DIRE for Diffusion-Generated Image Detectioncited by 386
  20. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  21. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  22. 2022MAGVIT: Masked Generative Video Transformercited by 341
  23. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  24. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  25. 2023GenImage: A Million-Scale Benchmark for Detecting AI-Generated Imagecited by 318
  26. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  27. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusioncited by 309
  28. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  29. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  30. 2023Text-to-3D using Gaussian Splattingcited by 299
  31. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 290
  32. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  33. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  34. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 281
  35. 2023Text-to-image Diffusion Models in Generative AI: A Surveycited by 264
  36. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  37. 2023MDTv2: Masked Diffusion Transformer is a Strong Image Synthesizercited by 260
  38. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258

50 most cited of 773 All 773 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation
  2. 2024Accurate Compression of Text-to-Image Diffusion Models via Vector Quantization
  3. 2025Diffusion Model Quantization: A Review
  4. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Models
  6. 2021Diffusion Autoencoders: Toward a Meaningful and Decodable Representation
  7. 2024DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation
  8. 2024VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion Transformers