mlsanityData to 14 Oct 2026

Enabling Approximate Joint Sampling in Diffusion LMs

arXiv 2509.22738 ↗First posted 25 Sep 2025cs.CL cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

In autoregressive language models, each token is sampled by conditioning on all the past tokens; the overall string has thus been sampled from the correct underlying joint distribution represented by the model. In contrast, masked diffusion language models generate text by unmasking tokens out of order and potentially in parallel. Generating an overall string sampled from the correct underlying joint distribution would (again) require exactly one token unmasking in every full-model forward pass. The more tokens unmasked in parallel, the further away the string is from the true joint; this can be seen in the resulting drop in accuracy (but, increase in speed). In this paper we devise a way to em approximately sample multiple tokens from the joint distribution in a single full-model forward pass; we do so by developing a new lightweight single-layer ``sampler" on top of an existing large diffusion LM. One forward pass of the full model can now be followed by multiple forward passes of only this sampler layer, to yield multiple unmasked tokens. Our sampler is trained to mimic exact joint sampling from the (frozen) full model. We show the effectiveness of our approximate joint sampling for both pretrained-only (Dream-7B-Base, Llada-7B-Base) and instruction-tuned (Dream-7B-Instruct, Dream-7B-Coder) models on language modeling and math & coding tasks. When four tokens are unmasked for each full-model denoising step, our sampling algorithm achieves a MAUVE score of 0.87 (vs marginal baseline of 0.31) with respect to the true joint distribution.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 27 of its 34 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

27 of 34 references.

References · 27

  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  3. 2025Large Language Diffusion Modelscited by 912
  4. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  5. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  6. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  7. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 626
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  9. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  10. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  11. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  12. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
15 more
  1. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 178
  3. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 112
  4. 2024Energy-Based Diffusion Language Models for Text Generationcited by 89
  5. 2025Path Planning for Masked Diffusion Model Samplingcited by 80
  6. 2024Learning Harmonized Representations for Speculative Samplingcited by 72
  7. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  8. 2024Think While You Generate: Discrete Diffusion with Planned Denoisingcited by 63
  9. 2024Discrete Copula Diffusioncited by 54
  10. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 47
  11. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 34
  12. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Maskingcited by 20
  13. 2021Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integralscited by 18
  14. 2025Anchored Diffusion Language Modelcited by 17
  15. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 30
  2. 2026Dependency-Guided Parallel Decoding in Discrete Diffusion Language Modelscited by 8
  3. 2026Locally Coherent Parallel Decoding in Diffusion Language Modelscited by 3
  4. 2026Fast-dLLM++: Fr'echet Profile Decoding for Faster Diffusion LLM Inferencecited by 2
  5. 2026Sangam: Efficiently Serving Diffusion LLMs with the AR Stackcited by 2
  6. 2026Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Modelscited by 1
  7. 2026The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Modelscited by 1
  8. 2026Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reportedcited by 0
  9. 2026Alpha Diffusion Language Models: Factorization Alone Is Not the Problemcited by 0
  10. 2026Intent-Triggered Anchoring for Multi-Token Format Units in Parallel Decoding of Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Models
  2. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
  3. 2025Optimal Inference Schedules for Masked Diffusion Models
  4. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Models
  5. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking
  6. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  7. 2026Adaptation to Intrinsic Dependence in Diffusion Language Models
  8. 2026Infinite Mask Diffusion for Few-Step Distillation