# Computational Tradeoffs in Image Synthesis: Diffusion, Masked-Token, and Next-Token Prediction

arXiv: https://arxiv.org/abs/2405.13218 · HTML: https://mlsanity.com/paper/2405.13218/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 21 May 2024.

## Builds on (most cited first, 24 of 24)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction](https://mlsanity.com/paper/2404.02905/) (2024)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors](https://mlsanity.com/paper/2203.13131/) (2022)
- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [Scaling Laws for Generative Mixed-Modal Language Models](https://mlsanity.com/paper/2301.03728/) (2023)
- [Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design](https://mlsanity.com/paper/2305.13035/) (2023)
- [Wuerstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.00637/) (2023)
- [Bigger is not Always Better: Scaling Properties of Latent Diffusion Models](https://mlsanity.com/paper/2404.01367/) (2024)

## Built on by (most cited first, 13 of 13)

- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models](https://mlsanity.com/paper/2501.01423/) (2025)
- [DiP: Taming Diffusion Models in Pixel Space](https://mlsanity.com/paper/2511.18822/) (2025)
- [DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation](https://mlsanity.com/paper/2410.08159/) (2024)
- [CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction](https://mlsanity.com/paper/2412.06782/) (2024)
- [When Worse is Better: Navigating the compression-generation tradeoff in visual tokenization](https://mlsanity.com/paper/2412.16326/) (2024)
- [Denoising with a Joint-Embedding Predictive Architecture](https://mlsanity.com/paper/2410.03755/) (2024)
- [[MASK] is All You Need](https://mlsanity.com/paper/2412.06787/) (2024)
- [L2P: Unlocking Latent Potential for Pixel Generation](https://mlsanity.com/paper/2605.12013/) (2026)
- [High-Resolution Image Synthesis via Next-Token Prediction](https://mlsanity.com/paper/2411.14808/) (2024)
- [Spanning Tree Autoregressive Visual Generation](https://mlsanity.com/paper/2511.17089/) (2025)
- [PuYun-LDM: A Latent Diffusion Model for High-Resolution Ensemble Weather Forecasts](https://mlsanity.com/paper/2602.11807/) (2026)
- [Geometry-Aware Representation Denoising for Robust Multi-view 3D Reconstruction](https://mlsanity.com/paper/2605.26230/) (2026)

## Abstract (arXiv)

Nearly every recent image synthesis approach, including diffusion, masked-token prediction, and next-token prediction, uses a Transformer network architecture. Despite this common backbone, there has been no direct, compute controlled comparison of how these approaches affect performance and efficiency. We analyze the scalability of each approach through the lens of compute budget measured in FLOPs. We find that token prediction methods, led by next-token prediction, significantly outperform diffusion on prompt following. On image quality, while next-token prediction initially performs better, scaling trends suggest it is eventually matched by diffusion. We compare the inference compute efficiency of each approach and find that next token prediction is by far the most efficient. Based on our findings we recommend diffusion for applications targeting image quality and low latency; and next-token prediction when prompt following or throughput is more important.