mlsanityData to 7 Oct 2026

Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models

arXiv 2606.10537 ↗First posted 9 Jun 2026cs.CLcited by 2

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion large language models (dLLMs) re-encode the entire prefix at every denoising step, causing recomputation that scales quadratically with context length and becomes prohibitive for long-context scenarios. We propose Prefilling-dLLM, a training-free prefill-decode disaggregation framework for dLLMs that partitions the prefix into N chunks, caches their KV representations once, and selects the top-K most relevant chunks with intra-chunk token sparsity for decoding, showing that sparse prefilling can outperform dense attention while reducing per-step complexity from quadratic in the full sequence length to quadratic only in the decode length. On LongBench and InfiniteBench, Prefilling-dLLM achieves state-of-the-art quality among dLLM acceleration methods, and an attention kernel that parallelizes decoding over the non-contiguously cached chunk KV yields 9.1--28.0x speedup at 8K--32K contexts. We further show that beginning-of-sequence tokens prepended to each chunk act as periodic attention anchors that eliminate the lost-in-the-middle phenomenon. Code is available at https://github.com/menik1126/Prefilling-dLLM.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 30 of its 36 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

30 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  2. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs17 Jun
  3. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  4. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  5. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 25

  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  4. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  6. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  7. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  8. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  9. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  10. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  11. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  12. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
13 more
  1. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  3. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  4. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  5. 2021Self-attention Does Not Need O(n2) Memorycited by 152
  6. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  7. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  8. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  9. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 64
  10. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  11. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 15
  12. 2025SPAD: Specialized Prefill and Decode Hardware for Disaggregated LLM Inferencecited by 6
  13. 2026Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scalecited by 1
  2. 2026Affix Cache for Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
  2. 2026PulseCol: Periodically Refreshed Column-Sparse Attention for Accelerating Diffusion Language Models
  3. 2026Retrofitting Linear Attention into Diffusion Language Models
  4. 2026Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Models
  5. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
  6. 2026ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping
  7. 2026CAI-DLLM: Convergence Aware Inference for Diffusion Language Models
  8. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques