# Enabling Approximate Joint Sampling in Diffusion LMs

arXiv: https://arxiv.org/abs/2509.22738 · HTML: https://mlsanity.com/paper/2509.22738/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 27 of 27)

- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models](https://mlsanity.com/paper/2309.12284/) (2023)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [Dream 7B: Diffusion Large Language Models](https://mlsanity.com/paper/2508.15487/) (2025)
- [Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding](https://mlsanity.com/paper/2505.22618/) (2025)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [Mercury: Ultra-Fast Language Models Based on Diffusion](https://mlsanity.com/paper/2506.17298/) (2025)
- [Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling](https://mlsanity.com/paper/2409.02908/) (2024)
- [Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference](https://mlsanity.com/paper/2508.02193/) (2025)
- [Accelerating Diffusion LLMs via Adaptive Parallel Decoding](https://mlsanity.com/paper/2506.00413/) (2025)
- [Energy-Based Diffusion Language Models for Text Generation](https://mlsanity.com/paper/2410.21357/) (2024)
- [Path Planning for Masked Diffusion Model Sampling](https://mlsanity.com/paper/2502.03540/) (2025)
- [Learning Harmonized Representations for Speculative Sampling](https://mlsanity.com/paper/2408.15766/) (2024)
- [Training and Inference on Any-Order Autoregressive Models the Right Way](https://mlsanity.com/paper/2205.13554/) (2022)
- [Think While You Generate: Discrete Diffusion with Planned Denoising](https://mlsanity.com/paper/2410.06264/) (2024)
- [Discrete Copula Diffusion](https://mlsanity.com/paper/2410.01949/) (2024)
- [Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potential](https://mlsanity.com/paper/2507.11851/) (2025)
- [Esoteric Language Models: A Family of Any-Order Diffusion LLMs](https://mlsanity.com/paper/2506.01928/) (2025)
- [Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking](https://mlsanity.com/paper/2505.18495/) (2025)
- [Divergence Frontiers for Generative Models: Sample Complexity, Quantization Effects, and Frontier Integrals](https://mlsanity.com/paper/2106.07898/) (2021)
- [Anchored Diffusion Language Model](https://mlsanity.com/paper/2505.18456/) (2025)
- [Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding](https://mlsanity.com/paper/2504.20456/) (2025)

## Built on by (most cited first, 10 of 10)

- [Learning Unmasking Policies for Diffusion Language Models](https://mlsanity.com/paper/2512.09106/) (2025)
- [Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models](https://mlsanity.com/paper/2604.02560/) (2026)
- [Locally Coherent Parallel Decoding in Diffusion Language Models](https://mlsanity.com/paper/2603.20216/) (2026)
- [Fast-dLLM++: Fr'echet Profile Decoding for Faster Diffusion LLM Inference](https://mlsanity.com/paper/2606.02955/) (2026)
- [Sangam: Efficiently Serving Diffusion LLMs with the AR Stack](https://mlsanity.com/paper/2607.04206/) (2026)
- [Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models](https://mlsanity.com/paper/2602.19619/) (2026)
- [The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Models](https://mlsanity.com/paper/2605.24697/) (2026)
- [Simple Diffusion Language Models Are More Effective Few-Step Generators Than Reported](https://mlsanity.com/paper/2609.33947/) (2026)
- [Alpha Diffusion Language Models: Factorization Alone Is Not the Problem](https://mlsanity.com/paper/2609.38066/) (2026)
- [Intent-Triggered Anchoring for Multi-Token Format Units in Parallel Decoding of Diffusion Language Models](https://mlsanity.com/paper/2610.16516/) (2026)

## Abstract (arXiv)

In autoregressive language models, each token is sampled by conditioning on all the past tokens; the overall string has thus been sampled from the correct underlying joint distribution represented by the model. In contrast, masked diffusion language models generate text by unmasking tokens out of order and potentially in parallel. Generating an overall string sampled from the correct underlying joint distribution would (again) require exactly one token unmasking in every full-model forward pass. The more tokens unmasked in parallel, the further away the string is from the true joint; this can be seen in the resulting drop in accuracy (but, increase in speed). In this paper we devise a way to em approximately sample multiple tokens from the joint distribution in a single full-model forward pass; we do so by developing a new lightweight single-layer ``sampler" on top of an existing large diffusion LM. One forward pass of the full model can now be followed by multiple forward passes of only this sampler layer, to yield multiple unmasked tokens. Our sampler is trained to mimic exact joint sampling from the (frozen) full model. We show the effectiveness of our approximate joint sampling for both pretrained-only (Dream-7B-Base, Llada-7B-Base) and instruction-tuned (Dream-7B-Instruct, Dream-7B-Coder) models on language modeling and math & coding tasks. When four tokens are unmasked for each full-model denoising step, our sampling algorithm achieves a MAUVE score of 0.87 (vs marginal baseline of 0.31) with respect to the true joint distribution.