mlsanityData to 14 Oct 2026

A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Models

arXiv 2508.08712 ↗First posted 12 Aug 2025cs.CL cs.AI cs.DCcited by 30

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

As text generation has become a core capability of modern Large Language Models (LLMs), it underpins a wide range of downstream applications. However, most existing LLMs rely on autoregressive (AR) generation, producing one token at a time based on previously generated context-resulting in limited generation speed due to the inherently sequential nature of the process. To address this challenge, an increasing number of researchers have begun exploring parallel text generation-a broad class of techniques aimed at breaking the token-by-token generation bottleneck and improving inference efficiency. Despite growing interest, there remains a lack of comprehensive analysis on what specific techniques constitute parallel text generation and how they improve inference performance. To bridge this gap, we present a systematic survey of parallel text generation methods. We categorize existing approaches into AR-based and Non-AR-based paradigms, and provide a detailed examination of the core techniques within each category. Following this taxonomy, we assess their theoretical trade-offs in terms of speed, quality, and efficiency, and examine their potential for combination and comparison with alternative acceleration strategies. Finally, based on our findings, we highlight recent advancements, identify open challenges, and outline promising directions for future research in parallel text generation. We have also created a GitHub repository for indexing relevant papers and open resources available at https://github.com/zhanglingzhe0820/Awesome-Parallel-Text-Generation.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 148 of its 270 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

148 of 270 references.

Closest ancestors · 27

Same direction, newest first

  1. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing8 Aug
  2. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference4 Aug
  3. 2025Diffusion Beats Autoregressive in Data-Constrained Settings21 Jul
  4. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  5. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  6. 2025The Diffusion Duality12 Jun
  7. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking30 May
  8. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusion27 May
  9. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decoding22 May
  10. 2025dKV-Cache: The Cache for Diffusion Language Models21 May
  11. 2025CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation20 May
  12. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
15 more
  1. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  2. 2025Large Language Diffusion Models14 Feb
  3. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions10 Feb
  4. 2025Path Planning for Masked Diffusion Model Sampling5 Feb
  5. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  6. 2024Scaling up Masked Diffusion Models on Text24 Oct
  7. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  8. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
  9. 2024Think While You Generate: Discrete Diffusion with Planned Denoising8 Oct
  10. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  11. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  12. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  13. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning23 Aug
  14. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  15. 2021Autoregressive Diffusion Models5 Oct

Other references · 121

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2023A Survey of Large Language Modelscited by 2,576
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2023Consistency Modelscited by 1,822
  12. 2024OpenAI o1 System Cardcited by 1,244
38 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  4. 2024DeepSeek-V3 Technical Reportcited by 811
  5. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  6. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  7. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  8. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  9. 2017Non-Autoregressive Neural Machine Translationcited by 463
  10. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  11. 2023Improved Techniques for Training Consistency Modelscited by 439
  12. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  13. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  14. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  15. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  16. 2024Simplifying, Stabilizing and Scaling Continuous-Time Consistency Modelscited by 283
  17. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 235
  18. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 195
  19. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  20. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  21. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 169
  22. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  23. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  24. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 161
  25. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  26. 2023REST: Retrieval-Based Speculative Decodingcited by 150
  27. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  28. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  29. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 121
  30. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 117
  31. 2023Online Speculative Decodingcited by 113
  32. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 112
  33. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 105
  34. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  35. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  36. 2020Fully Non-autoregressive Neural Machine Translation: Tricks of the Tradecited by 99
  37. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  38. 2019FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flowcited by 89

50 most cited of 121 All 121 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 20.

30 papers.

Sort:

By area:

  1. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  2. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 41
  3. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 31
  4. 2025Free Draft-and-Verification: Toward Lossless Parallel Decoding for Diffusion Large Language Modelscited by 19
  5. 2025ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decodingcited by 18
  6. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  7. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 17
  8. 2025Adaptive Root Cause Localization for Microservice Systems with Multi-Agent Recursion-of-Thoughtcited by 13
  9. 2025Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Modelcited by 13
  10. 2025Fast-Decoding Diffusion Language Models via Progress-Aware Confidence Schedulescited by 11
  11. 2026TDGNet: Hallucination Detection in Diffusion Language Models via Temporal Dynamic Graphscited by 8
  12. 2025On the Reasoning Abilities of Masked Diffusion Language Modelscited by 7
18 more
  1. 2026Streaming-dLLM: Accelerating Diffusion LLMs via Suffix Pruning and Dynamic Decodingcited by 7
  2. 2026SeeUPO: Sequence-Level Agentic-RL with Convergence Guaranteescited by 6
  3. 2026Efficient Failure Management for Multi-Agent Systems with Reasoning Trace Representationcited by 6
  4. 2026From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discoverycited by 6
  5. 2026Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Modelscited by 5
  6. 2026Towards Robust LLM Post-Training: Automatic Failure Management for Reinforcement Fine-Tuningcited by 3
  7. 2026dMoE: dLLMs with Learnable Block Expertscited by 2
  8. 2026Towards In-Depth Root Cause Localization for Microservices with Multi-Agent Recursion-of-Thoughtcited by 1
  9. 2026Continuous Language Diffusion as a Decoder-Interface Problemcited by 1
  10. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniquescited by 1
  11. 2026AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?cited by 0
  12. 2026Progressive Refinement Regulation for Accelerating Diffusion Language Model Decodingcited by 0
  13. 2026Regulating Branch Parallelism in LLM Servingcited by 0
  14. 2026GPO-V: Jailbreak Diffusion Vision Language Model by Global Probability Optimizationcited by 0
  15. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Modelscited by 0
  16. 2026HyperZip: Efficient Data Compression through Personalized Diffusion LLMs with Hypernetworkscited by 0
  17. 2026ALoDLM: Adaptively Looped Diffusion Language Modelscited by 0
  18. 2026Playing social deduction games with reinforcement fine-tuned large language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generation
  2. 2025FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Models
  3. 2024Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
  4. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
  5. 2024Energy-Based Diffusion Language Models for Text Generation
  6. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey
  7. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation
  8. 2025Alternatives To Next Token Prediction In Text Generation -- A Survey