mlsanityData to 7 Oct 2026

Attention Sinks in Diffusion Language Models

arXiv 2510.15731 ↗First posted 17 Oct 2025cs.CL cs.AIcited by 18

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Masked Diffusion Language Models (DLMs) have recently emerged as a promising alternative to traditional Autoregressive Models (ARMs). DLMs employ transformer encoders with bidirectional attention, enabling parallel token generation while maintaining competitive performance. Although their efficiency and effectiveness have been extensively studied, the internal mechanisms that govern DLMs remain largely unexplored. In this work, we conduct an empirical analysis of DLM attention patterns, focusing on the attention sinking phenomenon, an effect previously observed in various transformer-based architectures. Our findings reveal that DLMs also exhibit attention sinks, but with distinct characteristics. First, unlike in ARMs, the sink positions in DLMs tend to shift throughout the generation process, displaying a dynamic behaviour. Second, while ARMs are highly sensitive to the removal of attention sinks, DLMs remain robust: masking sinks leads to only a minor degradation in performance. These results provide new insights into the inner workings of diffusion-based language models and highlight fundamental differences in how they allocate and utilize attention compared to autoregressive models.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 28 of its 48 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

28 of 48 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Any-Order Flexible Length Masked Diffusion31 Aug
  2. 2025Dream 7B: Diffusion Large Language Models21 Aug
  3. 2025Implicit Search via Discrete Diffusion: A Study on Chess27 Feb
  4. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  5. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  7. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  8. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  9. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  10. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  12. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
8 more
  1. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  2. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 163
  3. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  4. 2025Why do LLMs attend to the first token?cited by 122
  5. 2024Transformers need glasses! Information over-squashing in language taskscited by 87
  6. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  7. 2024Spectral Filters, Dark Signals, and Attention Sinkscited by 54
  8. 2025What are you sinking? A geometric approach on attention sinkcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 17.

18 papers.

Sort:

By area:

  1. 2026DAWN: Dependency-Aware Fast Inference for Diffusion LLMscited by 12
  2. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 10
  3. 2025Masks Can Be Distracting: On Context Comprehension in Diffusion Language Modelscited by 7
  4. 2026Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusingcited by 6
  5. 2026Revealing the Attention Floating Mechanism in Masked Diffusion Modelscited by 4
  6. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  7. 2026One Token Is Enough: Improving Diffusion Language Models with a Sink Tokencited by 2
  8. 2026A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMscited by 2
  9. 2026Attention Sinks in Diffusion Transformers: A Causal Analysiscited by 2
  10. 2026Registers Matter for Pixel-Space Diffusion Transformerscited by 2
  11. 2026Sink-Aware Pruning for Diffusion Language Modelscited by 1
  12. 2026Adaptive Guidance for Retrieval-Augmented Masked Diffusion Modelscited by 1
6 more
  1. 2026Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing preventioncited by 1
  2. 2026Text Template Tokens Are Implicit Semantic Registers in Diffusion Transformerscited by 1
  3. 2026Order Matters: LVLMs as Judges for Temporal Reasoning in Image Sequencescited by 1
  4. 2026PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Modelscited by 0
  5. 2026Subliminal Clocks: Latent Time Modelling in Diffusion Language Modelscited by 0
  6. 2026SoloQ: Calibration-Free Quantization for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Empirical Analysis of Decoding Biases in Masked Diffusion Models
  2. 2025A Survey on Diffusion Language Models
  3. 2026Induction in Both Directions: A Mechanistic Analysis of In-Context Learning in Masked Diffusion Language Models
  4. 2025Why mask diffusion does not work
  5. 2026PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
  6. 2026dQwen3.5: Hybrid-Attention Diffusion Language Models
  7. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  8. 2024Simple and Effective Masked Diffusion Language Models