mlsanityData to 14 Oct 2026

TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models

arXiv 2605.24292 ↗First posted 22 May 2026cs.LGcited by 2

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Log-likelihood is a standard metric for evaluating generative models. Unfortunately, in contrast to autoregressive models (ARMs), discrete diffusion models generally do not admit exact computation of this quantity. Existing evaluations, therefore, rely on the evidence lower bound (ELBO), leaving unclear how much higher the true value may be. We address this by introducing the Tangent Upper Bound on Evidence (TUBE), a variational upper bound on log-likelihood that admits an unbiased Monte Carlo estimator. Our TUBE extends across latent-variable models, including masked diffusion models (MDMs), any-order ARMs (AO-ARMs), and block variants of both. Applied to block MDMs and block AO-ARMs, TUBE reveals our key empirical finding that these models lie strictly below the exact ARM baseline, showing that ARMs still dominate in likelihood.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 44 of its 51 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

44 of 51 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Generative Frontiers: Why Evaluation Matters for Diffusion Language Models3 Apr
  2. 2026DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking2 Mar
  3. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B10 Dec
  4. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models10 Oct
  5. 2025FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models24 Sep
  6. 2025Dream 7B: Diffusion Large Language Models21 Aug
  7. 2025Efficient Perplexity Bound and Ratio Matching in Discrete Diffusion Language Models6 Jul
  8. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  9. 2025Large Language Diffusion Models14 Feb
  10. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  11. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  12. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
3 more
  1. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  2. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  3. 2021Autoregressive Diffusion Models5 Oct

Other references · 29

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  9. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  10. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  11. 2015Importance Weighted Autoencoderscited by 456
  12. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
17 more
  1. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  2. 2016R'enyi Divergence Variational Inferencecited by 104
  3. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 83
  4. 2013A Deep and Tractable Density Estimatorcited by 68
  5. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  6. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 64
  7. 2025GenMol: A Drug Discovery Generalist with Discrete Diffusioncited by 50
  8. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  9. 2025Learning-Order Autoregressive Models with Application to Molecular Graph Generationcited by 29
  10. 2019The Thermodynamic Variational Objectivecited by 26
  11. 2021Discovering Non-monotonic Autoregressive Orderings with Variational Inferencecited by 15
  12. 2026Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMDcited by 12
  13. 2019Stochastic Variational Inference via Upper Boundcited by 8
  14. 2025Learning Flexible Forward Trajectories for Masked Molecular Diffusioncited by 7
  15. 2022Bounding Evidence and Estimating Log-Likelihood in VAEcited by 6
  16. 2023Discriminator Guidance for Autoregressive Diffusion Modelscited by 5
  17. 2025Information-Theoretic Discrete Diffusioncited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Modelscited by 0
  2. 2026IDRF: Inverse-Distilled Reward Fine-tuning of Masked Discrete Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimal Inference Schedules for Masked Diffusion Models
  2. 2026Vocabulary-size-independent Convergence of Discrete Diffusion Models: adjoint equations induce the right space
  3. 2023Likelihood-Based Diffusion Language Models
  4. 2026From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion
  5. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  6. 2026Optimizers for Diffusion Models: A Controlled Benchmark
  7. 2026Adaptation to Intrinsic Dependence in Diffusion Language Models
  8. 2026Fork-dLLM: Avoiding the Flexibility Trap in Diffusion Language Models