mlsanityData to 14 Oct 2026

Self-Speculative Masked Diffusions

arXiv 2510.03929 ↗First posted 4 Oct 2025stat.ML cs.LGcited by 11

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

We present self-speculative masked diffusions, a new class of masked diffusion generative models for discrete data that require significantly fewer function evaluations to generate samples. Standard masked diffusion models predict factorized logits over currently masked positions. A number of masked positions are then sampled, however, the factorization approximation means that sampling too many positions in one go leads to poor sample quality. As a result, many simulation steps and therefore neural network function evaluations are required to generate high-quality data. We reduce the computational burden by generating non-factorized predictions over masked positions. This is achieved by modifying the final transformer attention mask from non-causal to causal, enabling draft token generation and parallel validation via a novel, model-integrated speculative sampling mechanism. This results in a non-factorized predictive distribution over masked positions in a single forward pass. We apply our method to GPT2 scale text modelling and protein sequence generation, finding that we can achieve a ~2x reduction in the required number of network forward passes relative to standard masked diffusion models.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 31 of its 37 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

31 of 37 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Path Planning for Masked Diffusion Model Sampling5 Feb
  2. 2024Energy-Based Diffusion Language Models for Text Generation28 Oct
  3. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  4. 2024Discrete Copula Diffusion2 Oct
  5. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  6. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  7. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  8. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  9. 2021Autoregressive Diffusion Models5 Oct

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  10. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  11. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  12. 2023Protein Design with Guided Discrete Diffusioncited by 157
10 more
  1. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 98
  2. 2024RandAR: Decoder-only Autoregressive Visual Generation in Random Orderscited by 83
  3. 2013A Deep and Tractable Density Estimatorcited by 68
  4. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  5. 2024DPLM-2: A Multimodal Diffusion Protein Language Modelcited by 64
  6. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  7. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  8. 2024Distillation of Discrete Diffusion through Dimensional Correlationscited by 48
  9. 2024σ-GPTs: A New Approach to Autoregressive Modelscited by 20
  10. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 7.

11 papers.

  1. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 30
  2. 2025Dream-VL & Dream-VLA: Open Vision-Language and Vision-Language-Action Models with Diffusion Language Model Backbonecited by 18
  3. 2025DEER: Draft with Diffusion, Verify with Autoregressive Modelscited by 9
  4. 2026Fixed-Point Masked Generative Modelingcited by 5
  5. 2026Locally Coherent Parallel Decoding in Diffusion Language Modelscited by 3
  6. 2026Forward-Free Diffusion Language Models with BPTT-Free Looped Refinementcited by 2
  7. 2026Accelerating Speculative Diffusions via Block Verificationcited by 2
  8. 2026Generative Modeling in Protein Design: Neural Representations, Conditional Generation, and Evaluation Standardscited by 1
  9. 2026Multi-Mask Diffusion Language Models for Few-Step Generationcited by 1
  10. 2025Latent-Augmented Discrete Diffusion Modelscited by 0
  11. 2026Factorization-Error-Free Discrete Diffusion Language Model via Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking
  2. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generator
  3. 2025Error Bounds and Optimal Schedules for Masked Diffusions with Factorized Approximations
  4. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction
  5. 2025Why Masking Diffusion Works: Condition on the Jump Schedule for Improved Discrete Diffusion
  6. 2025Parallel Sampling from Masked Diffusion Models via Conditional Independence Testing
  7. 2025Accelerated Diffusion Models via Speculative Sampling
  8. 2026DualDiffusion: A Speculative Decoding Strategy for Masked Diffusion Models