mlsanityData to 14 Oct 2026

Scaling Diffusion Language Models via Adaptation from Autoregressive Models

arXiv 2410.17891 ↗First posted 23 Oct 2024cs.CLcited by 274

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, current DLMs have been studied at a smaller scale compared to their AR counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. Given the prevalence of open-source AR language models, we propose adapting these models to build text diffusion models. We demonstrate connections between AR and diffusion modeling objectives and introduce a simple continual pre-training approach for training diffusion models. Through systematic evaluation on language modeling, reasoning, and commonsense benchmarks, we show that we can convert AR models ranging from 127M to 7B parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training. Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts. We release a suite of DLMs (127M-355M-7B) capable of generating fluent text, performing in-context learning, filling in the middle without prompt re-ordering, and following instructions https://github.com/HKUNLP/DiffuLLaMA.

Key facts. Cited by 274 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 84 of its 98 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

84 of 98 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  2. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  3. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  4. 2024Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models12 Feb
  5. 2023DiffuSeq-v2: Bridging Discrete and Continuous Text Spaces for Accelerated Seq2Seq Diffusion Models9 Oct
  6. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning23 Aug
  7. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model5 Jun
  8. 2023Likelihood-Based Diffusion Language Models30 May
  9. 2023Diffusion Models for Non-autoregressive Text Generation: A Survey12 Mar
  10. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  11. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise22 Dec
  12. 2022Continuous diffusion for categorical data28 Nov
4 more
  1. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  2. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  3. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct
  4. 2021Autoregressive Diffusion Models5 Oct

Other references · 68

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  11. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  12. 2020Denoising Diffusion Implicit Modelscited by 9,781
38 more
  1. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  2. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  13. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  14. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  15. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  16. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  17. 2022Emergent Abilities of Large Language Modelscited by 2,512
  18. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  19. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  20. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  21. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  22. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  23. 2021Variational Diffusion Modelscited by 1,153
  24. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  25. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  26. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  27. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  28. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  29. 2023StarCoder: may the source be with you!cited by 799
  30. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  31. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  32. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  33. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  34. 2017Non-Autoregressive Neural Machine Translationcited by 463
  35. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  36. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 392
  37. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  38. 2022Efficient Training of Language Models to Fill in the Middlecited by 255

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 125; 2026: 147.

274 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 912
  2. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  3. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  4. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  5. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  6. 2024Scaling up Masked Diffusion Models on Textcited by 199
  7. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  8. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  9. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  10. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 166
  11. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 143
  12. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
38 more
  1. 2025Dimple: Discrete Diffusion Multimodal Large Language Model with Parallel Decodingcited by 101
  2. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  3. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  4. 2025Path Planning for Masked Diffusion Model Samplingcited by 80
  5. 2025A Survey on Diffusion Language Modelscited by 78
  6. 2025Diffusion Language Models Know the Answer Before Decodingcited by 70
  7. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  8. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 57
  9. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 54
  10. 2025Any-Order Flexible Length Masked Diffusioncited by 51
  11. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 47
  12. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  13. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 42
  14. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  15. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 41
  16. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40
  17. 2025Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Processcited by 39
  18. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modelingcited by 36
  19. 2025Beyond Fixed: Training-Free Variable-Length Denoising for Diffusion Large Language Modelscited by 34
  20. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
  21. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 33
  22. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 33
  23. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Modelscited by 32
  24. 2025Coevolutionary Continuous Discrete Diffusion: Make Your Diffusion Language Model a Latent Reasonercited by 32
  25. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  26. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  27. 2025LLaDA-VLA: Vision Language Diffusion Action Modelscited by 30
  28. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  29. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decodingcited by 27
  30. 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decodingcited by 26
  31. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  32. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inferencecited by 24
  33. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMscited by 24
  34. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 24
  35. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Modelscited by 23
  36. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 23
  37. 2025From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Modelscited by 23
  38. 2025MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Controlcited by 21

50 most cited of 274 All 274 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing Diffusion Language Models with Autoregressive Post-Training Weights
  2. 2024Transfer Learning for Text Diffusion Models
  3. 2023A Survey of Diffusion Models in Natural Language Processing
  4. 2024Energy-Based Diffusion Language Models for Text Generation
  5. 2026Top 10 Open Challenges Steering the Future of Diffusion Language Model and Its Variants
  6. 2023Diffusion Language Models Generation Can Be Halted Early
  7. 2025Diffusion Beats Autoregressive in Data-Constrained Settings
  8. 2022Self-conditioned Embedding Diffusion for Text Generation