mlsanityData to 7 Oct 2026

What Does a Discrete Diffusion Model Learn?

arXiv 2607.05381 ↗First posted 6 Jul 2026cs.LG cs.AI cs.CLcited by 1

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

What does a discrete diffusion model learn: a denoiser, a score ratio, or a bridge plug-in predictor? At the level of jump rates, these are one object in different coordinates, and reading a neural network in the wrong coordinate changes the process being trained and sampled. Starting with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process, boundary terms included, we prove the Oracle Distance theorem: the negative ELBO is exactly equal to the data entropy plus the path KL from the oracle reverse process to the learned one, not merely a bound. Its unique optimizer is therefore the conditional expectation of the true reverse jump rate given the current noisy state, and its irreducible cost is the rate at which the forward process Zt destroys information about the clean data Z0, -tfracddtI(Z0; Zt), so every noising process shares the same best achievable negative ELBO: the data entropy. For sequences with token-factorizing noise, the oracle projection yields three exact coordinates for the optimizer: denoiser, cavity (bridge plug-in), and score, with closed-form conversions among them. This framework identifies which law each loss in the literature actually optimizes, recovering MDM, UDM, SEDD, and GIDD as special cases; explains why denoiser and cavity coincide for masked diffusion but not for uniform diffusion; proves that a denoiser parameterization makes the uniform ELBO diverge at initialization while the bridge plug-in stays finite; and calibrates ELBO implementations exactly at initialization. Every identity is verified numerically, without approximation, on an exactly solvable model.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 5 of its 44 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

5 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation21 May
  2. 2025Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Bounds24 Nov

Other references · 3

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2026Scaling Beyond Masked Diffusion Language Modelscited by 22
  3. 2026Noise Scheduling as Information-Guided Allocation in Diffusion Trainingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Neural Continuous-Time Markov Chain: Discrete Diffusion via Decoupled Jump Timing and Direction
  2. 2026Conservation Laws for Diffusion Models
  3. 2026Mean-to-Score Discrete Diffusion: Posterior-Mean Denoisers for Score Entropy
  4. 2022A Continuous Time Framework for Discrete Denoising Models
  5. 2026Sticky Jump Diffusions: A Unifying View of Masked, Continuous, and Hybrid Diffusion
  6. 2025Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction
  7. 2026Efficient Sampling with Discrete Diffusion Models: Sharp and Adaptive Guarantees
  8. 2025Information-Theoretic Discrete Diffusion