mlsanityData to 14 Oct 2026

Variational Autoencoding Discrete Diffusion with Enhanced Dimensional Correlations Modeling

arXiv 2505.17384 ↗First posted 23 May 2025cs.LG cs.CV stat.MLcited by 14

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

Discrete diffusion models have recently shown great promise for modeling complex discrete data, with masked diffusion models (MDMs) offering a compelling trade-off between quality and generation speed. MDMs denoise by progressively unmasking multiple dimensions from an all-masked input, but their performance can degrade when using few denoising steps due to limited modeling of inter-dimensional dependencies. In this paper, we propose Variational Autoencoding Discrete Diffusion (VADD), a novel framework that enhances discrete diffusion with latent variable modeling to implicitly capture correlations among dimensions. By introducing an auxiliary recognition model, VADD enables stable training via variational lower bounds maximization and amortized inference over the training set. Our approach retains the efficiency of traditional MDMs while significantly improving sample quality, especially when the number of denoising steps is small. Empirical results on 2D toy data, pixel-level image generation, and text generation demonstrate that VADD consistently outperforms MDM baselines in sample quality with few denoising steps.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 37 of its 48 references that are in the map. First posted on arXiv 23 May 2025.

Builds on

37 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Distillation of Discrete Diffusion through Dimensional Correlations11 Oct
  2. 2022Score-based Continuous-time Discrete Diffusion Models30 Nov
  3. 2022A Continuous Time Framework for Discrete Denoising Models30 May
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Models20 Dec
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spaces7 Jul
  6. 2021Variational Diffusion Models1 Jul
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equations26 Nov
  8. 2020Denoising Diffusion Probabilistic Models19 Jun

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2022Video Diffusion Modelscited by 2,305
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  5. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  6. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  7. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  9. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 640
  10. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 637
  11. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  12. 2015Generating Sentences from a Continuous Spacecited by 612
17 more
  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  2. 2017Non-Autoregressive Neural Machine Translationcited by 463
  3. 2015Importance Weighted Autoencoderscited by 456
  4. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  5. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  6. 2019Understanding and Improving Layer Normalizationcited by 217
  7. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Modelscited by 214
  8. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  9. 2018Fast Decoding in Sequence Models using Discrete Latent Variablescited by 104
  10. 2024Energy-Based Diffusion Language Models for Text Generationcited by 89
  11. 2018Semi-Amortized Variational Autoencoderscited by 83
  12. 2017Improved Variational Autoencoders for Text Modeling using Dilated Convolutionscited by 81
  13. 2016PixelVAE: A Latent Variable Model for Natural Imagescited by 70
  14. 2020A Contrastive Learning Approach for Training Variational Autoencoder Priorscited by 66
  15. 2024Discrete Copula Diffusioncited by 54
  16. 2018Avoiding Latent Variable Collapse With Generative Skip Modelscited by 47
  17. 2021OmniNet: Omnidirectional Representations from Transformerscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2026IDLM: Inverse-distilled Diffusion Language Modelscited by 8
  2. 2026Unifying Masked Diffusion Models with Various Generation Orders and Beyondcited by 6
  3. 2026DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modelingcited by 4
  4. 2026Accelerating Discrete Diffusion Models with Parallel-In-Time Samplingcited by 4
  5. 2026Plug-and-Play Guidance for Discrete Diffusion Models via Gradient-Informed Logit Correctioncited by 3
  6. 2026Simplex Relaxation for Discrete Diffusioncited by 2
  7. 2026CRoCoDiL: Continuous and Robust Conditioned Diffusion for Languagecited by 1
  8. 2026Infinite Mask Diffusion for Few-Step Distillationcited by 1
  9. 2026Learned Relay Representations for Forward-Thinking Discrete Diffusion Modelscited by 1
  10. 2026Tensor-Train Joint Modeling for Few-Step Discrete Diffusioncited by 1
  11. 2025Latent-Augmented Discrete Diffusion Modelscited by 0
  12. 2026Understanding and Accelerating the Training of Masked Diffusion Language Modelscited by 0
2 more
  1. 2026A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Modelscited by 0
  2. 2026E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Variational Diffusion Auto-encoder: Latent Space Extraction from Pre-trained Diffusion Models
  2. 2026Forward-Learned Discrete Diffusion: Learning how to noise to denoise faster
  3. 2022Understanding Diffusion Models: A Unified Perspective
  4. 2024DisCo-Diff: Enhancing Continuous Diffusion Models with Discrete Latents
  5. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking
  6. 2026SRA 2: Variational Autoencoder Self-Representation Alignment for Efficient Diffusion Training
  7. 2025Variational Masked Diffusion Models
  8. 2025Latent Diffusion Models with Masked AutoEncoders