mlsanityData to 7 Oct 2026

Dynamic Chunking for Diffusion Language Models

arXiv 2605.15676 ↗First posted 15 May 2026cs.CLcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Block discrete diffusion language models factorize a sequence autoregressively over fixed-size positional blocks, decoupling within-block parallel denoising from across-block conditioning. We argue that this rigid partition wastes structure already present in the sequence: blocks defined by position rather than by content separate semantically coherent tokens and group unrelated ones together. We introduce the Dynamic Chunking Diffusion Model (DCDM), which replaces positional blocks with content-defined semantic chunks. At its core is Chunking Attention, a differentiable layer that routes tokens into K clusters parameterized by learnable subspaces and shaped end-to-end by the diffusion objective. The resulting cluster assignments induce a chunk-causal attention mask under which a discrete diffusion denoiser factorizes the sequence likelihood autoregressively over semantic chunks, strictly generalizing block discrete diffusion. On downstream benchmarks at parameter scales up to 1.5B, DCDM consistently improves over both unstructured and positional-block diffusion baselines, with the advantage stable across scales and visible early in training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 39 of its 45 references that are in the map. First posted on arXiv 15 May 2026.

Builds on

39 of 45 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Dream 7B: Diffusion Large Language Models21 Aug
  2. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  3. 2025Large Language Diffusion Models14 Feb
  4. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  5. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  6. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  7. 2021Autoregressive Diffusion Models5 Oct

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2023Qwen Technical Reportcited by 3,273
20 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2020Aligning AI With Shared Human Valuescited by 706
  10. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  11. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  12. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  13. 2022Efficient Training of Language Models to Fill in the Middlecited by 248
  14. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  15. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 191
  16. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  17. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 30
  18. 2025Attention-based clusteringcited by 3
  19. 2023Deep Structure and Attention Aware Subspace Clusteringcited by 1
  20. 2025A mini-batch training strategy for deep subspace clustering networkscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models
  2. 2026Fixed State, Long Reach: What a Constant-Size Cache Buys Block Diffusion at Scale
  3. 2026Optimizers for Diffusion Models: A Controlled Benchmark
  4. 2026Diffusion In Diffusion: Reclaiming Global Coherence in Semi-Autoregressive Diffusion
  5. 2026Retrofitting Linear Attention into Diffusion Language Models
  6. 2025Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding
  7. 2026Window-Diffusion: Accelerating Diffusion Language Model Inference with Windowed Token Pruning and Caching
  8. 2026SemBlock: Semantic Boundary Dynamic Blocks for Diffusion LLMs