# Global Context with Discrete Diffusion in Vector Quantised Modelling for Image Generation

arXiv: https://arxiv.org/abs/2112.01799 · HTML: https://mlsanity.com/paper/2112.01799/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 40 of its 44 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Pixel Recurrent Neural Networks](https://mlsanity.com/paper/1601.06759/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)

## Built on by (most cited first, 30 of 40)

- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [A Survey on Generative Diffusion Model](https://mlsanity.com/paper/2209.02646/) (2022)
- [A Reparameterized Discrete Diffusion Model for Text Generation](https://mlsanity.com/paper/2302.05737/) (2023)
- [MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation](https://mlsanity.com/paper/2209.09002/) (2022)
- [A Survey on Diffusion Models for Time Series and Spatio-Temporal Data](https://mlsanity.com/paper/2404.18886/) (2024)
- [Online Clustered Codebook](https://mlsanity.com/paper/2307.15139/) (2023)
- [How Discrete and Continuous Diffusion Meet: Comprehensive Analysis of Discrete Diffusion Models via a Stochastic Integral Framework](https://mlsanity.com/paper/2410.03601/) (2024)
- [Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms](https://mlsanity.com/paper/2502.00234/) (2025)
- [Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation](https://mlsanity.com/paper/2206.07771/) (2022)
- [Unified Discrete Diffusion for Simultaneous Vision-Language Generation](https://mlsanity.com/paper/2211.14842/) (2022)
- [Stochastic Segmentation with Conditional Categorical Diffusion Models](https://mlsanity.com/paper/2303.08888/) (2023)
- [Diffusion Probabilistic Models for Scene-Scale 3D Categorical Data](https://mlsanity.com/paper/2301.00527/) (2023)
- [Diff-RNTraj: A Structure-aware Diffusion Model for Road Network-constrained Trajectory Generation](https://mlsanity.com/paper/2402.07369/) (2024)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Diffusion Model is a Good Pose Estimator from 3D RF-Vision](https://mlsanity.com/paper/2403.16198/) (2024)
- [A Survey on Diffusion Models for Recommender Systems](https://mlsanity.com/paper/2409.05033/) (2024)
- [DLM-One: Diffusion Language Models for One-Step Sequence Generation](https://mlsanity.com/paper/2506.00290/) (2025)
- [Learning to Jump: Thinning and Thickening Latent Counts for Generative Modeling](https://mlsanity.com/paper/2305.18375/) (2023)
- [Beta Diffusion](https://mlsanity.com/paper/2309.07867/) (2023)
- [Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation](https://mlsanity.com/paper/2406.01561/) (2024)
- [Dimensionality-Varying Diffusion Process](https://mlsanity.com/paper/2211.16032/) (2022)
- [Cocktail: Mixing Multi-Modality Controls for Text-Conditional Image Generation](https://mlsanity.com/paper/2306.00964/) (2023)
- [Glauber Generative Model: Discrete Diffusion Models via Binary Classification](https://mlsanity.com/paper/2405.17035/) (2024)
- [Text-guided Explorable Image Super-resolution](https://mlsanity.com/paper/2403.01124/) (2024)
- [Spiking-Diffusion: Vector Quantized Discrete Diffusion Model with Spiking Neural Networks](https://mlsanity.com/paper/2308.10187/) (2023)
- [One More Step: A Versatile Plug-and-Play Module for Rectifying Diffusion Schedule Flaws and Enhancing Low-Frequency Controls](https://mlsanity.com/paper/2311.15744/) (2023)
- [Lformer: Text-to-Image Generation with L-shape Block Parallel Decoding](https://mlsanity.com/paper/2303.03800/) (2023)
- [Topic-VQ-VAE: Leveraging Latent Codebooks for Flexible Topic-Guided Document Generation](https://mlsanity.com/paper/2312.11532/) (2023)
- [Watermarking Discrete Diffusion Language Models](https://mlsanity.com/paper/2511.02083/) (2025)

## Abstract (arXiv)

The integration of Vector Quantised Variational AutoEncoder (VQ-VAE) with autoregressive models as generation part has yielded high-quality results on image generation. However, the autoregressive models will strictly follow the progressive scanning order during the sampling phase. This leads the existing VQ series models to hardly escape the trap of lacking global information. Denoising Diffusion Probabilistic Models (DDPM) in the continuous domain have shown a capability to capture the global context, while generating high-quality images. In the discrete state space, some works have demonstrated the potential to perform text generation and low resolution image generation. We show that with the help of a content-rich discrete visual codebook from VQ-VAE, the discrete diffusion model can also generate high fidelity images with global context, which compensates for the deficiency of the classical autoregressive model along pixel space. Meanwhile, the integration of the discrete VAE with the diffusion model resolves the drawback of conventional autoregressive models being oversized, and the diffusion model which demands excessive time in the sampling process when generating images. It is found that the quality of the generated images is heavily dependent on the discrete visual codebook. Extensive experiments demonstrate that the proposed Vector Quantised Discrete Diffusion Model (VQ-DDM) is able to achieve comparable performance to top-tier methods with low complexity. It also demonstrates outstanding advantages over other vectors quantised with autoregressive models in terms of image inpainting tasks without additional training.