mlsanityData to 14 Oct 2026

Your Absorbing Discrete Diffusion Secretly Models the Bayesian Posterior

arXiv 2507.07586 ↗First posted 10 Jul 2025cs.CL cs.AI cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Discrete diffusion language models learn to reconstruct text from randomly masked inputs, yet under mild assumptions their denoiser already implements the exact Bayesian posterior over the original tokens. We prove that the expected denoiser output under the forward corruption distribution recovers the true posterior, and that a simple Monte Carlo estimator converges to this posterior at rate O(1/sqrt(K)) with finite-sample concentration bounds. Building on this insight, we introduce an inference-time ensemble that runs K independent denoising passes and aggregates both posterior means and variances without any extra training. On WikiText-2, our MC-marginal sampler recovers the analytic lambda-DCE zero-shot perplexity (approximately 39) to within a few points at K=128, and its per-token variance shows a strong rank correlation with reconstruction error (Spearman rho = 0.996). This cost-proportional procedure yields calibrated uncertainty estimates and a direct trade-off between compute and posterior fidelity in discrete diffusion LMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 7 of its 8 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

7 of 8 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun

Other references · 6

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  5. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Support Before Frequency in Discrete Diffusion
  2. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  3. 2026Attention-Discounted Adaptive Sampler for Masked Diffusion Language Models
  4. 2026Consistent Diffusion Language Models
  5. 2025On the ε-Free Inference Complexity of Absorbing Discrete Diffusion
  6. 2026Bayesian Entropy-based Reordering for Calibrated Diffusion Language Models
  7. 2026CoDR: Training-Free Confidence-Drift Remasking for Diffusion Language Models
  8. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inference