mlsanityData to 7 Oct 2026

Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?

arXiv 2602.23225 ↗First posted 26 Feb 2026cs.CL cs.AIcited by 11

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

also in 2 more
Abstract (arXiv)

Diffusion Language Models (DLMs) are often advertised as enabling parallel token generation, yet practical fast DLMs frequently converge to left-to-right, autoregressive (AR)-like decoding dynamics. In contrast, genuinely non-AR generation is promising because it removes AR's sequential bottleneck, better exploiting parallel hardware to reduce synchronization/communication overhead and improve latency scaling with output length. We argue that a primary driver of AR-like decoding is a mismatch between DLM objectives and the highly sequential structure of widely used training data, including standard pretraining corpora and long chain-of-thought (CoT) supervision. Motivated by this diagnosis, we propose NAP (Non-Autoregressive Parallel DLMs), a proof-of-concept, data-centric approach that better aligns supervision with non-AR parallel decoding. NAP curates examples as multiple independent reasoning trajectories and couples them with a parallel-forced decoding strategy that encourages multi-token parallel updates. Across math reasoning benchmarks, NAP yields stronger performance under parallel decoding than DLMs trained on standard long CoT data, with gains growing as parallelism increases. Our results suggest that revisiting data and supervision is a principled direction for mitigating AR-like behavior and moving toward genuinely non-autoregressive parallel generation in DLMs. Our code is available at https://github.com/pixeli99/NAP.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 38 of its 46 references that are in the map. First posted on arXiv 26 Feb 2026.

Builds on

38 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning16 Apr

Other references · 37

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  10. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  12. 2025Large Language Diffusion Modelscited by 872
25 more
  1. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Datacited by 606
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  4. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  5. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  6. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  7. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 381
  8. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 371
  9. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  10. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  11. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  12. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  13. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  14. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  15. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  16. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  17. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 111
  18. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  19. 2025Path Planning for Masked Diffusion Model Samplingcited by 79
  20. 2025Diffusion Language Models Know the Answer Before Decodingcited by 68
  21. 2024Speculative Diffusion Decoding: Accelerating Language Generation through Diffusioncited by 57
  22. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  23. 2025ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Computecited by 36
  24. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 29
  25. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Modelscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Fast-dLLM++: Fr'echet Profile Decoding for Faster Diffusion LLM Inferencecited by 2
  2. 2026Rethinking the Generation Order of Block Diffusion Language Modelscited by 2
  3. 2026TRIMS: Trajectory-Ranked Instruction Masked Supervision for Diffusion Language Modelscited by 1
  4. 2026Visual-Redundancy-Controlled Parallel Decoding for Diffusion-Based Multimodal Large Language Modelscited by 1
  5. 2026Ripple-Pivot Search: Active Parallel Decoding for Diffusion Large Language Modelscited by 1
  6. 2026Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Modelscited by 1
  7. 2026Path-Dependent Denoising: A Non-Conservative Field Perspective on Order Collapse in Diffusion Language Modelscited by 0
  8. 2026PSD: Pushing the Pareto Frontier of Diffusion LLMs via Parallel Speculative Decodingcited by 0
  9. 2026From Interface to Inference: Eliciting Any-Order Inference from Any-Order Modelscited by 0
  10. 2026Early-Bird Decoding: Accelerating Diffusion LLMs with Learnable Block Sizes and Parallel Samplingcited by 0
  11. 2026d-OPD: Future-Aware On-Policy Distillation for Block Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Locally Coherent Parallel Decoding in Diffusion Language Models
  2. 2026Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding
  3. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decoding
  4. 2026Mean-Field Parallel Decoding for Discrete Diffusion Language Models
  5. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inference
  6. 2026Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow
  7. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
  8. 2026SimSD: Simple Speculative Decoding in Diffusion Language Models