# Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes

arXiv: https://arxiv.org/abs/2111.12701 · HTML: https://mlsanity.com/paper/2111.12701/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 78 of its 97 references that are in the map.
- First posted on arXiv 24 Nov 2021.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 78)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Image Super-Resolution via Iterative Refinement](https://mlsanity.com/paper/2104.07636/) (2021)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)

## Built on by (most cited first, 30 of 71)

- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Blended Latent Diffusion](https://mlsanity.com/paper/2206.02779/) (2022)
- [A Survey on Video Diffusion Models](https://mlsanity.com/paper/2310.10647/) (2023)
- [Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models](https://mlsanity.com/paper/2306.04675/) (2023)
- [A Reparameterized Discrete Diffusion Model for Text Generation](https://mlsanity.com/paper/2302.05737/) (2023)
- [Collaborative Diffusion for Multi-Modal Face Generation and Editing](https://mlsanity.com/paper/2304.10530/) (2023)
- [CityDreamer: Compositional Generative Model of Unbounded 3D Cities](https://mlsanity.com/paper/2309.00610/) (2023)
- [Diffusion Models for Graphs Benefit From Discrete State Spaces](https://mlsanity.com/paper/2210.01549/) (2022)
- [A Text-guided Protein Design Framework](https://mlsanity.com/paper/2302.04611/) (2023)
- [Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2304.03322/) (2023)
- [Towards Accurate Image Coding: Improved Autoregressive Image Generation with Dynamic Vector Quantization](https://mlsanity.com/paper/2305.11718/) (2023)
- [On the Design Fundamentals of Diffusion Models: A Survey](https://mlsanity.com/paper/2306.04542/) (2023)
- [DiffBP: Generative Diffusion of 3D Molecules for Target Protein Binding](https://mlsanity.com/paper/2211.11214/) (2022)
- [Text2Performer: Text-Driven Human Video Generation](https://mlsanity.com/paper/2304.08483/) (2023)
- [Fast Solvers for Discrete Diffusion Models: Theory and Applications of High-Order Algorithms](https://mlsanity.com/paper/2502.00234/) (2025)
- [Diffusion Facial Forgery Detection](https://mlsanity.com/paper/2401.15859/) (2024)
- [RIGID: A Training-free and Model-Agnostic Framework for Robust AI-Generated Image Detection](https://mlsanity.com/paper/2405.20112/) (2024)
- [AbDiffuser: Full-Atom Generation of in vitro Functioning Antibodies](https://mlsanity.com/paper/2308.05027/) (2023)
- [Text2Human: Text-Driven Controllable Human Image Generation](https://mlsanity.com/paper/2205.15996/) (2022)
- [Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer](https://mlsanity.com/paper/2206.04452/) (2022)
- [DiffBEV: Conditional Diffusion Model for Bird's Eye View Perception](https://mlsanity.com/paper/2303.08333/) (2023)
- [Text2Light: Zero-Shot Text-Driven HDR Panorama Generation](https://mlsanity.com/paper/2209.09898/) (2022)
- [Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning](https://mlsanity.com/paper/2310.12952/) (2023)
- [∞-Diff: Infinite Resolution Diffusion with Subsampled Mollified States](https://mlsanity.com/paper/2303.18242/) (2023)
- [Binary Latent Diffusion](https://mlsanity.com/paper/2304.04820/) (2023)
- [Efficient Image Generation with Variadic Attention Heads](https://mlsanity.com/paper/2211.05770/) (2022)
- [Explaining generative diffusion models via visual analysis for interpretable decision-making process](https://mlsanity.com/paper/2402.10404/) (2024)
- [Compositional Generative Model of Unbounded 4D Cities](https://mlsanity.com/paper/2501.08983/) (2025)
- [Discrete Diffusion Probabilistic Models for Symbolic Music Generation](https://mlsanity.com/paper/2305.09489/) (2023)
- [Be More Diverse than the Most Diverse: Optimal Mixtures of Generative Models via Mixture-UCB Bandit Algorithms](https://mlsanity.com/paper/2412.17622/) (2024)

## Abstract (arXiv)

Whilst diffusion probabilistic models can generate high quality image content, key limitations remain in terms of both generating high-resolution imagery and their associated high computational requirements. Recent Vector-Quantized image models have overcome this limitation of image resolution but are prohibitively slow and unidirectional as they generate tokens via element-wise autoregressive sampling from the prior. By contrast, in this paper we propose a novel discrete diffusion probabilistic model prior which enables parallel prediction of Vector-Quantized tokens by using an unconstrained Transformer architecture as the backbone. During training, tokens are randomly masked in an order-agnostic manner and the Transformer learns to predict the original tokens. This parallelism of Vector-Quantized token prediction in turn facilitates unconditional generation of globally consistent high-resolution and diverse imagery at a fraction of the computational expense. In this manner, we can generate image resolutions exceeding that of the original training set samples whilst additionally provisioning per-image likelihood estimates (in a departure from generative adversarial approaches). Our approach achieves state-of-the-art results in terms of Density (LSUN Bedroom: 1.51; LSUN Churches: 1.12; FFHQ: 1.20) and Coverage (LSUN Bedroom: 0.83; LSUN Churches: 0.73; FFHQ: 0.80), and performs competitively on FID (LSUN Bedroom: 3.64; LSUN Churches: 4.07; FFHQ: 6.11) whilst offering advantages in terms of both computation and reduced training set requirements.