# Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling

arXiv: https://arxiv.org/abs/2505.17384 · HTML: https://mlsanity.com/paper/2505.17384/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models.
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 23 May 2025.

## Placed in

- Diffusion models / [Diffusion foundations and sampling](https://mlsanity.com/diffusion/diffusion-foundations-and-sampling/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Variational Diffusion Models](https://mlsanity.com/paper/2107.00630/) (2021)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [Improving Variational Inference with Inverse Autoregressive Flow](https://mlsanity.com/paper/1606.04934/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Non-Autoregressive Neural Machine Translation](https://mlsanity.com/paper/1711.02281/) (2017)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [A Continuous Time Framework for Discrete Denoising Models](https://mlsanity.com/paper/2205.14987/) (2022)
- [Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data](https://mlsanity.com/paper/2406.03736/) (2024)
- [One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling](https://mlsanity.com/paper/1312.3005/) (2013)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models](https://mlsanity.com/paper/2211.15029/) (2022)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [Score-based Continuous-time Discrete Diffusion Models](https://mlsanity.com/paper/2211.16750/) (2022)
- [Fast Decoding in Sequence Models using Discrete Latent Variables](https://mlsanity.com/paper/1803.03382/) (2018)
- [Energy-Based Diffusion Language Models for Text Generation](https://mlsanity.com/paper/2410.21357/) (2024)
- [Semi-Amortized Variational Autoencoders](https://mlsanity.com/paper/1802.02550/) (2018)

## Built on by (most cited first, 14 of 14)

- [IDLM: Inverse-distilled Diffusion Language Models](https://mlsanity.com/paper/2602.19066/) (2026)
- [Unifying Masked Diffusion Models with Various Generation Orders and Beyond](https://mlsanity.com/paper/2602.02112/) (2026)
- [DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling](https://mlsanity.com/paper/2605.23605/) (2026)
- [Accelerating Discrete Diffusion Models with Parallel-In-Time Sampling](https://mlsanity.com/paper/2607.00773/) (2026)
- [Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correction](https://mlsanity.com/paper/2606.06303/) (2026)
- [Simplex Relaxation for Discrete Diffusion](https://mlsanity.com/paper/2608.10615/) (2026)
- [CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language](https://mlsanity.com/paper/2603.20210/) (2026)
- [Infinite Mask Diffusion for Few-Step Distillation](https://mlsanity.com/paper/2605.10518/) (2026)
- [Learned Relay Representations for Forward-Thinking Discrete Diffusion Models](https://mlsanity.com/paper/2605.22967/) (2026)
- [Tensor-Train Joint Modeling for Few-Step Discrete Diffusion](https://mlsanity.com/paper/2607.03788/) (2026)
- [Latent-Augmented Discrete Diffusion Models](https://mlsanity.com/paper/2510.18114/) (2025)
- [Understanding and Accelerating the Training of Masked Diffusion Language Models](https://mlsanity.com/paper/2605.13026/) (2026)
- [A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models](https://mlsanity.com/paper/2609.06324/) (2026)
- [E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models](https://mlsanity.com/paper/2609.37533/) (2026)

## Abstract (arXiv)

Discrete diffusion models have recently shown great promise for modeling complex discrete data, with masked diffusion models (MDMs) offering a compelling trade-off between quality and generation speed. MDMs denoise by progressively unmasking multiple dimensions from an all-masked input, but their performance can degrade when using few denoising steps due to limited modeling of inter-dimensional dependencies. In this paper, we propose Variational Autoencoding Discrete Diffusion (VADD), a novel framework that enhances discrete diffusion with latent variable modeling to implicitly capture correlations among dimensions. By introducing an auxiliary recognition model, VADD enables stable training via variational lower bounds maximization and amortized inference over the training set. Our approach retains the efficiency of traditional MDMs while significantly improving sample quality, especially when the number of denoising steps is small. Empirical results on 2D toy data, pixel-level image generation, and text generation demonstrate that VADD consistently outperforms MDM baselines in sample quality with few denoising steps.