mlsanityData to 14 Oct 2026

DPad: Efficient Diffusion Language Models with Suffix Dropout

arXiv 2508.14148 ↗First posted 19 Aug 2025cs.CL cs.LGcited by 39

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion-based Large Language Models (dLLMs) parallelize text generation by framing decoding as a denoising process, but suffer from high computational overhead since they predict all future suffix tokens at each step while retaining only a small fraction. We propose Diffusion Scratchpad (DPad), a training-free method that restricts attention to a small set of nearby suffix tokens, preserving fidelity while eliminating redundancy. DPad integrates two strategies: (i) a sliding window, which maintains a fixed-length suffix window, and (ii) distance-decay dropout, which deterministically removes distant suffix tokens before attention computation. This simple design is compatible with existing optimizations such as prefix caching and can be implemented with only a few lines of code. Comprehensive evaluations across multiple benchmarks on LLaDA-1.5 and Dream models demonstrate that DPad delivers up to 61.4× speedup over vanilla dLLMs while maintaining comparable accuracy, highlighting its potential for efficient and scalable long-sequence inference. Our code is available at https://github.com/Crys-Chen/DPad.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 28 of its 32 references that are in the map. First posted on arXiv 19 Aug 2025.

Builds on

28 of 32 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  2. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  3. 2025Large Language Diffusion Models14 Feb
  4. 2024Scaling up Masked Diffusion Models on Text24 Oct
  5. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2020Longformer: The Long-Document Transformercited by 3,413
11 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  4. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  5. 2017Non-Autoregressive Neural Machine Translationcited by 463
  6. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  7. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
  8. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  9. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  10. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 65
  11. 2025Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principlescited by 45

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 26.

39 papers.

Sort:

By area:

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  2. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  3. 2025Diffusion Language Models Know the Answer Before Decodingcited by 70
  4. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  5. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 42
  6. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 31
  7. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  8. 2025FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Modelscited by 17
  9. 2025CreditDecoding: Accelerating Parallel Decoding in Diffusion Large Language Models with Trace Creditcited by 13
  10. 2025How Efficient Are Diffusion Language Models? A Critical Examination of Efficiency Evaluation Practicescited by 10
  11. 2025Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Modelscited by 9
  12. 2026TEAM: Temporal-Spatial Consistency Guided Expert Activation for MoE Diffusion Language Model Accelerationcited by 8
27 more
  1. 2026Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decodingcited by 7
  2. 2026ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skippingcited by 6
  3. 2025Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Modelscited by 5
  4. 2026Improving Variable-Length Generation in Diffusion Language Models via Length Regularizationcited by 4
  5. 2026Breaking Block Boundaries: Anchor-based History-stable Decoding for Diffusion Large Language Modelscited by 4
  6. 2025A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Modelscited by 3
  7. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Modelscited by 3
  8. 2026Consistent Diffusion Language Modelscited by 2
  9. 2026GroupKV: Hierarchical KV Cache Management for Long-Context Diffusion LLM Inferencecited by 2
  10. 2026STDec: Spatio-Temporal Stability Guided Decoding for dLLMscited by 1
  11. 2026DARE: Diffusion Language Model Activation Reuse for Efficient Inferencecited by 1
  12. 2026Elastic-dLLM: Position Preserving Context Compression and Augmentation of Diffusion LLMscited by 1
  13. 2026The Path Matters: Learning a Token-Commitment Policy for Diffusion Language Modelscited by 1
  14. 2026Optimus: Elastic Decoding for Efficient Diffusion LLM Servingcited by 1
  15. 2025STaRR: Spatial-Temporal Token-Dynamics-Aware Responsive Remasking for Diffusion Language Modelscited by 0
  16. 2026Progressive Refinement Regulation for Accelerating Diffusion Language Model Decodingcited by 0
  17. 2026Beyond Scattered Acceptance: Fast and Coherent Inference for DLMs via Longest Stable Prefixescited by 0
  18. 2026Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Modelscited by 0
  19. 2026Understanding Evaluation Illusion in Diffusion Large Language Modelscited by 0
  20. 2026LaCache: Exact Caching and Precision-Adaptive Inference for Diffusion Large Language Modelscited by 0
  21. 2026DC-Leap: Training-Free Acceleration of dLLMs via Draft-Guided Contiguous Leaping Decodingcited by 0
  22. 2026Commit Locally, Exit Globally: Coordinating Adaptive Sampling and Early Exit in Diffusion Language Modelscited by 0
  23. 2026Accelerating Diffusion Language Models via Structured Suffix Modelingcited by 0
  24. 2026Affix Cache for Diffusion Large Language Modelscited by 0
  25. 2026Trajectory-Level Speculative Decoding for Diffusion Language Modelscited by 0
  26. 2026Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Samplingcited by 0
  27. 2026HaloBlock: Faster and Better Decoding for Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
  2. 2026Diffusion Language Models: An Experimental Analysis
  3. 2026DAWN: Dependency-Aware Fast Inference for Diffusion LLMs
  4. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  5. 2026Beyond Masks: Efficient, Flexible Diffusion Language Models via Deletion-Insertion Processes
  6. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion
  7. 2026DOS: Dependency-Oriented Sampler for Masked Diffusion Language Models
  8. 2025Dream 7B: Diffusion Large Language Models