mlsanityData to 7 Oct 2026

Revealing the Attention Floating Mechanism in Masked Diffusion Models

arXiv 2601.07894 ↗First posted 12 Jan 2026cs.LG cs.AIcited by 4

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked diffusion models (MDMs), which leverage bidirectional attention and a denoising process, are narrowing the performance gap with autoregressive models (ARMs). However, their internal attention mechanisms remain under-explored. This paper investigates the attention behaviors in MDMs, revealing the phenomenon of Attention Floating. Unlike ARMs, where attention converges to a fixed sink, MDMs exhibit dynamic, dispersed attention anchors that shift across denoising steps and layers. Further analysis reveals its Shallow Structure-Aware, Deep Content-Focused attention mechanism: shallow layers utilize floating tokens to build a global structural framework, while deeper layers allocate more capability toward capturing semantic content. Empirically, this distinctive attention pattern provides a mechanistic explanation for the strong in-context learning capabilities of MDMs, allowing them to double the performance compared to ARMs in knowledge-intensive tasks. All codes and datasets are available at https://github.com/NEUIR/Attention-Floating.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 26 of its 30 references that are in the map. First posted on arXiv 12 Jan 2026.

Builds on

26 of 30 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025TiDAR: Think in Diffusion, Talk in Autoregression12 Nov
  2. 2025Diffuse Thinking: Exploring Diffusion Language Models as Efficient Thought Proposers for Reasoning31 Oct
  3. 2025Soft-Masked Diffusion Language Models20 Oct
  4. 2025Attention Sinks in Diffusion Language Models17 Oct
  5. 2025Training Optimal Large Diffusion Language Models28 Sep
  6. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs17 Jun
  7. 2025Smoothie: Smoothing Diffusion on Token Embeddings for Text Generation24 May
  8. 2025TESS 2: A Large-Scale Generalist Diffusion Language Model19 Feb
  9. 2024TEncDM: Understanding the Properties of the Diffusion Model in the Space of Language Model Encodings29 Feb
  10. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  11. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov

Other references · 15

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  6. 2022In-context Learning and Induction Headscited by 930
  7. 2020Quantifying Attention Flow in Transformerscited by 759
  8. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  9. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  10. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  11. 2024RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewardscited by 33
  12. 2022Composable Text Controls in Latent Space with ODEscited by 28
3 more
  1. 2025Dissecting and Mitigating Diffusion Bias via Mechanistic Interpretabilitycited by 26
  2. 2025Emergence and Evolution of Interpretable Concepts in Diffusion Modelscited by 20
  3. 2025Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026One Token Is Enough: Improving Diffusion Language Models with a Sink Tokencited by 2
  2. 2026Adaptive Guidance for Retrieval-Augmented Masked Diffusion Modelscited by 1
  3. 2026Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformerscited by 1
  4. 2026Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Empirical Analysis of Decoding Biases in Masked Diffusion Models
  2. 2026Attention Sinks in Diffusion Transformers: A Causal Analysis
  3. 2025SparseD: Sparse Attention for Diffusion Language Models
  4. 2025Bringing Stability to Diffusion: Decomposing and Reducing Variance of Training Masked Diffusion Models
  5. 2026PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
  6. 2026Beyond Fully Random Masking: Attention-Guided Denoising and Optimization for Diffusion Language Models
  7. 2026Retrofitting Linear Attention into Diffusion Language Models
  8. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking