mlsanityData to 14 Oct 2026

Prefilling-dLLM v2: Learned Chunk Routing for Long-Context Diffusion Language Models

arXiv 2610.17868 ↗First posted 13 Oct 2026cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion language models apply bidirectional attention over the whole context at every denoising step, so a longer context makes each step substantially more expensive. Sparsifying only at inference leaves the pretrained weights unchanged, so the model is never trained under the sparse pattern it runs with. We propose Prefilling-dLLM v2, which continues training a diffusion language model so that it routes with its native mask token. One mask is appended at the end of each chunk, and a learned role offset marks it as a routing slot. On every fourth layer, a query keeps a bidirectional local window and the top-K chunks scored from that mask; the other layers use a bidirectional sliding window. Starting from Dream-v0-Base-7B extended to 16K with YaRN, we compare Prefilling-dLLM v2 with bidirectional NSA, DSA, and SWA under the same layout, key budget, and dense initialization, using the same Fast-dLLM decoder. With the YaRN factor scaled to the evaluation length, Prefilling-dLLM v2 leads the other sparse methods on all three RULER tasks from 16K to 128K. Its mean accuracy at 128K is 7.4 points above NSA, the next sparse model, while DSA and SWA remain near zero. At the same length, decoding is 172x faster than with full dense attention.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 32 of its 34 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

32 of 34 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Prefilling-dLLM: Predictive Prefilling for Long-Context Inference in Diffusion Language Models9 Jun
  2. 2025SparseD: Sparse Attention for Diffusion Language Models28 Sep
  3. 2025Dream 7B: Diffusion Large Language Models21 Aug
  4. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMs17 Jun
  5. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  6. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  7. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  8. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  9. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 23

  1. 2020Longformer: The Long-Document Transformercited by 3,413
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  9. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  11. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
11 more
  1. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  2. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  4. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 198
  6. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  7. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  8. 2025Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Evictioncited by 65
  9. 2024UNComp: Can Matrix Entropy Uncover Sparsity? -- A Compressor Design from an Uncertainty-Aware Perspectivecited by 15
  10. 2025ParallelComp: Parallel Long-Context Compressor for Length Extrapolationcited by 14
  11. 2026Hierarchical Sparse Attention Done Right: Toward Infinite Context Modelingcited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Dynamic Chunking for Diffusion Language Models
  2. 2026Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decoding
  3. 2026CAI-DLLM: Convergence Aware Inference for Diffusion Language Models
  4. 2026Retrofitting Linear Attention into Diffusion Language Models
  5. 2026Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
  6. 2026TimeROME-DLM: Temporal Causal Tracing and Low-Rank Inference-Time Knowledge Editing for Masked Diffusion Language Models
  7. 2026Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
  8. 2026ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping