mlsanityData to 7 Oct 2026

Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models

arXiv 2605.29607 ↗First posted 28 May 2026cs.LGcited by 2

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Masked diffusion language models (MDLMs) enable parallel decoding by predicting all masked positions at each denoising step, yet existing training-free samplers usually decide which positions to commit at token-level granularity. We revisit this granularity and observe that reliable predictions often emerge as contiguous high-confidence spans, suggesting that the unit of parallel commitment can be larger than a single token. We first group adjacent high-confidence candidates into confidence-induced clusters (CICs) as span-level update units. We then use self-attention maps from the same forward pass to estimate inter-cluster dependencies, enabling conflict-aware selection of mutually compatible CICs for parallel commitment. This yields CLAD (Cluster-Level Attention-Guided Decoding), a training-free cluster-level decoder for MDLMs. Experiments on LLaDA and Dream model families across four reasoning and code-generation benchmarks show that CLAD achieves 1.77x--8.47x speedups over Vanilla decoding while maintaining broadly comparable task accuracy in most settings.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 24 of its 31 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

24 of 31 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Models7 Nov
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025Theoretical Benefit and Limitation of Diffusion Language Model13 Feb
  4. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  5. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  6. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2023Qwen Technical Reportcited by 3,273
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  11. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  12. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
6 more
  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  2. 2022Concrete Score Matching: Generalized Score Matching for Discrete Datacited by 144
  3. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  4. 2025Accelerating Diffusion LLM Inference via Local Determinism Propagationcited by 21
  5. 2026DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMscited by 13
  6. 2026R2-dLLM: Accelerating Diffusion Large Language Models via Spatio-Temporal Redundancy Reductioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026One Latent, Many Tokens: Jointly Learning Compressed Embeddings for Efficient Language Diffusioncited by 0
  2. 2026d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reliable Parallel Decoding in Masked Diffusion Language Models
  2. 2025Beyond Static Cutoffs: One-Shot Dynamic Thresholding for Diffusion Language Models
  3. 2026LEAP: Unlocking dLLM Parallelism via Lookahead Early-Convergence Token Detection
  4. 2026Mean-Field Parallel Decoding for Discrete Diffusion Language Models
  5. 2026DMax: Aggressive Parallel Decoding for dLLMs
  6. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
  7. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
  8. 2026Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Sampling