mlsanityData to 7 Oct 2026

Empirical Analysis of Decoding Biases in Masked Diffusion Models

arXiv 2508.13021 ↗First posted 18 Aug 2025cs.AI cs.CLcited by 19

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes are available at https://github.com/NEUIR/Uncode.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 26 of its 38 references that are in the map. First posted on arXiv 18 Aug 2025.

Builds on

26 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding22 May
  2. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  3. 2024Scaling up Masked Diffusion Models on Text24 Oct
  4. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  5. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  6. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey12 Mar

Other references · 20

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2024Qwen2 Technical Reportcited by 2,076
  8. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  9. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  12. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
8 more
  1. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  2. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  3. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  4. 2025On the Emergence of Position Bias in Transformerscited by 57
  5. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 54
  6. 2025Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMscited by 45
  7. 2024Transformers Can Navigate Mazes With Multi-Step Predictioncited by 11
  8. 2024Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 16.

19 papers.

Sort:

By area:

  1. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 29
  2. 2025d2Cache: Accelerating Diffusion-Based LLMs via Dual Adaptive Cachingcited by 24
  3. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decodingcited by 16
  4. 2026Omni-Diffusion: Unified Multimodal Understanding and Generation with Masked Discrete Diffusioncited by 9
  5. 2026TABES: Trajectory-Aware Backward-on-Entropy Steering for Masked Diffusion Modelscited by 3
  6. 2026Swordsman: Entropy-Driven Adaptive Block Partition for Efficient Diffusion Language Modelscited by 3
  7. 2026Tuning the Implicit Regularizer of Masked Diffusion Language Models: Enhancing Generalization via Insights from k-Paritycited by 1
  8. 2026AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?cited by 1
  9. 2026Lookahead Path Likelihood Optimization for Diffusion LLMscited by 1
  10. 2026Insertion Based Sequence Generation with Learnable Order Dynamicscited by 1
  11. 2026Stability-Weighted Decoding for Diffusion Language Modelscited by 1
  12. 2026Consistent Diffusion Language Modelscited by 1
7 more
  1. 2026Scheduling Thoughts: Learning the Order of Thought in Diffusion Language Modelscited by 1
  2. 2026When Confidence Misleads: Suffix Anchoring and Anchor-Proximity Confidence Modulation for Diffusion Language Modelscited by 0
  3. 2026Subliminal Clocks: Latent Time Modelling in Diffusion Language Modelscited by 0
  4. 2026DC-Leap: Training-Free Acceleration of dLLMs via Draft-Guided Contiguous Leaping Decodingcited by 0
  5. 2026Context-Aware Cluster Decoding: Semantic Anchor-Driven Coherence in dMLLMscited by 0
  6. 2026Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Modelscited by 0
  7. 2026Information-Guided Frontier Decoding: Contextual Utility-Driven Commitment in dMLLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Revealing the Attention Floating Mechanism in Masked Diffusion Models
  2. 2025Attention Sinks in Diffusion Language Models
  3. 2026Attention Sinks in Diffusion Transformers: A Causal Analysis
  4. 2025SparseD: Sparse Attention for Diffusion Language Models
  5. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Models
  6. 2026Retrofitting Linear Attention into Diffusion Language Models
  7. 2026Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models
  8. 2025Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models