mlsanityData to 7 Oct 2026

Sink-Aware Pruning for Diffusion Language Models

arXiv 2602.17664 ↗First posted 19 Feb 2026cs.CL cs.AI cs.LGcited by 1

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion Language Models (DLMs) incur high inference cost due to iterative denoising, motivating efficient pruning. Existing pruning heuristics largely inherited from autoregressive (AR) LLMs, typically preserve attention sink tokens because AR sinks serve as stable global anchors. We show that this assumption does not hold for DLMs: the attention-sink position exhibits substantially higher variance over the full generation trajectory (measured by how the dominant sink locations shift across timesteps), indicating that sinks are often transient and less structurally essential than in AR models. Based on this observation, we propose bf textttSink-Aware Pruning, which automatically identifies and prunes unstable sinks in DLMs (prior studies usually keep sinks for AR LLMs). Without retraining, our method achieves a better quality-efficiency trade-off and outperforms strong prior pruning baselines under matched compute. Our code is available at https://github.com/VILA-Lab/Sink-Aware-Pruning.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 27 of its 38 references that are in the map. First posted on arXiv 19 Feb 2026.

Builds on

27 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Diffusion Language Models are Super Data Learners5 Nov
  2. 2025Attention Sinks in Diffusion Language Models17 Oct
  3. 2025A Survey on Diffusion Language Models14 Aug
  4. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  5. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  7. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 20

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
8 more
  1. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  2. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  3. 2019Importance Estimation for Neural Network Pruningcited by 531
  4. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  5. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  6. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  7. 2025Why do LLMs attend to the first token?cited by 122
  8. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 10

Similar, not cited

Close in wording, no citation either way

  1. 2026One Token Is Enough: Improving Diffusion Language Models with a Sink Token
  2. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  3. 2026Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
  4. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  5. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  6. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
  7. 2026Path-Dependent Denoising: A Non-Conservative Field Perspective on Order Collapse in Diffusion Language Models
  8. 2026CAI-DLLM: Convergence Aware Inference for Diffusion Language Models