# ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs

arXiv: https://arxiv.org/abs/2510.04767 · HTML: https://mlsanity.com/paper/2510.04767/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion LLM and MoE inference, one of 12 directions in Efficient LLM inference and serving.
- Builds on 42 of its 58 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Placed in

- Efficient LLM inference and serving / [Diffusion LLM and MoE inference](https://mlsanity.com/inference/diffusion-llm-and-moe-inference/)

## Builds on (most cited first, 30 of 42)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Structured Denoising Diffusion Models in Discrete State-Spaces](https://mlsanity.com/paper/2107.03006/) (2021)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Simple and Effective Masked Diffusion Language Models](https://mlsanity.com/paper/2406.07524/) (2024)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution](https://mlsanity.com/paper/2310.16834/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Simplified and Generalized Masked Diffusion for Discrete Data](https://mlsanity.com/paper/2406.04329/) (2024)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding](https://mlsanity.com/paper/2505.22618/) (2025)
- [Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models](https://mlsanity.com/paper/2503.09573/) (2025)
- [LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models](https://mlsanity.com/paper/2505.19223/) (2025)
- [MMaDA: Multimodal Large Diffusion Language Models](https://mlsanity.com/paper/2505.15809/) (2025)
- [Mercury: Ultra-Fast Language Models Based on Diffusion](https://mlsanity.com/paper/2506.17298/) (2025)
- [Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions](https://mlsanity.com/paper/2502.06768/) (2025)
- [Remasking Discrete Diffusion Models with Inference-Time Scaling](https://mlsanity.com/paper/2503.00307/) (2025)
- [DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation](https://mlsanity.com/paper/2506.20639/) (2025)
- [d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning](https://mlsanity.com/paper/2504.12216/) (2025)
- [Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference](https://mlsanity.com/paper/2508.02193/) (2025)

## Built on by (most cited first, 30 of 43)

- [LLaDA2.1: Speeding Up Text Diffusion via Token Editing](https://mlsanity.com/paper/2602.08676/) (2026)
- [Categorical Flow Maps](https://mlsanity.com/paper/2602.12233/) (2026)
- [d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation](https://mlsanity.com/paper/2601.07568/) (2026)
- [Optimal Inference Schedules for Masked Diffusion Models](https://mlsanity.com/paper/2511.04647/) (2025)
- [CDLM: Consistency Diffusion Language Models For Faster Sampling](https://mlsanity.com/paper/2511.19269/) (2025)
- [DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMs](https://mlsanity.com/paper/2603.12996/) (2026)
- [Stable-DiffCoder: Pushing the Frontier of Code Diffusion Large Language Model](https://mlsanity.com/paper/2601.15892/) (2026)
- [Corrective Diffusion Language Models](https://mlsanity.com/paper/2512.15596/) (2025)
- [Parallelism and Generation Order in Masked Diffusion Language Models: Limits Today, Potential Tomorrow](https://mlsanity.com/paper/2601.15593/) (2026)
- [Why Diffusion Language Models Struggle with Truly Parallel (Non-Autoregressive) Decoding?](https://mlsanity.com/paper/2602.23225/) (2026)
- [How to Train Your Latent Diffusion Language Model Jointly With the Latent Space](https://mlsanity.com/paper/2605.07933/) (2026)
- [Posterior Refinement: Fast Language Generation via Any-Order Flow Maps](https://mlsanity.com/paper/2606.24773/) (2026)
- [Generation Order and Parallel Decoding in Masked Diffusion Models: An Information-Theoretic Perspective](https://mlsanity.com/paper/2602.00286/) (2026)
- [Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs](https://mlsanity.com/paper/2512.20573/) (2025)
- [Locally Coherent Parallel Decoding in Diffusion Language Models](https://mlsanity.com/paper/2603.20216/) (2026)
- [Lost in Diffusion: Uncovering Hallucination Patterns and Failure Modes in Diffusion Large Language Models](https://mlsanity.com/paper/2604.10556/) (2026)
- [LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models](https://mlsanity.com/paper/2604.12056/) (2026)
- [TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM](https://mlsanity.com/paper/2605.09536/) (2026)
- [Cluster-Level Attention-Guided Parallel Decoding for Masked Diffusion Language Models](https://mlsanity.com/paper/2605.29607/) (2026)
- [Fast-dLLM++: Fr'echet Profile Decoding for Faster Diffusion LLM Inference](https://mlsanity.com/paper/2606.02955/) (2026)
- [Self-Generated Error Training for Token Editing in Diffusion Language Models](https://mlsanity.com/paper/2606.17175/) (2026)
- [Don't Commit Alone: Joint Token Commitment in Diffusion Language Models](https://mlsanity.com/paper/2607.04469/) (2026)
- [BackPlay: Head-Only Look-Back Self-Correction for Diffusion Language Models](https://mlsanity.com/paper/2601.06428/) (2026)
- [TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs](https://mlsanity.com/paper/2601.20357/) (2026)
- [Thinking Out of Order: When Output Order Stops Reflecting Reasoning Order in Diffusion Language Models](https://mlsanity.com/paper/2601.22035/) (2026)
- [Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models](https://mlsanity.com/paper/2602.19619/) (2026)
- [Remask, Don't Replace: Token-to-Mask Refinement in Diffusion Language Models](https://mlsanity.com/paper/2604.18738/) (2026)
- [EPIC: Efficient and Parallel Inference under CFG Constraints for Diffusion Language Models](https://mlsanity.com/paper/2606.00722/) (2026)
- [Follow the Latent Roadmap: Navigating Revocable Decoding for Diffusion LLMs with Anchor Tokens](https://mlsanity.com/paper/2606.16847/) (2026)
- [Polestar: Drift-Aware Cache Calibration and Token Commitment for Efficient Inference of Diffusion LLMs](https://mlsanity.com/paper/2607.14107/) (2026)

## Abstract (arXiv)

While most autoregressive LLMs are constrained to one-by-one decoding, diffusion LLMs (dLLMs) have attracted growing interest for their potential to dramatically accelerate inference through parallel decoding. Despite this promise, the conditional independence assumption in dLLMs causes parallel decoding to ignore token dependencies, inevitably degrading generation quality when these dependencies are strong. However, existing works largely overlook these inherent challenges, and evaluations on standard benchmarks (e.g., math and coding) are not sufficient to capture the quality degradation caused by parallel decoding. To address this gap, we first provide an information-theoretic analysis of parallel decoding. We then conduct case studies on analytically tractable synthetic list operations from both data distribution and decoding strategy perspectives, offering quantitative insights that highlight the fundamental limitations of parallel decoding. Building on these insights, we propose ParallelBench, the first benchmark specifically designed for dLLMs, featuring realistic tasks that are trivial for humans and autoregressive LLMs yet exceptionally challenging for dLLMs under parallel decoding. Using ParallelBench, we systematically analyze both dLLMs and autoregressive LLMs, revealing that: (i) dLLMs under parallel decoding can suffer dramatic quality degradation in real-world scenarios, and (ii) current parallel decoding strategies struggle to adapt their degree of parallelism based on task difficulty, thus failing to achieve meaningful speedup without compromising quality. Our findings underscore the pressing need for innovative decoding methods that can overcome the current speed-quality trade-off. We release our benchmark to help accelerate the development of truly efficient dLLMs.