mlsanityData to 14 Oct 2026

Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning

arXiv 2308.12219 ↗First posted 23 Aug 2023cs.CL cs.AI cs.LGcited by 39

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

The recent surge of generative AI has been fueled by the generative power of diffusion probabilistic models and the scalable capabilities of large language models. Despite their potential, it remains elusive whether diffusion language models can solve general language tasks comparable to their autoregressive counterparts. This paper demonstrates that scaling diffusion models w.r.t. data, sizes, and tasks can effectively make them strong language learners. We build competent diffusion language models at scale by first acquiring knowledge from massive data via masked language modeling pretraining thanks to their intrinsic connections. We then reprogram pretrained masked language models into diffusion language models via diffusive adaptation, wherein task-specific finetuning and instruction finetuning are explored to unlock their versatility in solving general language tasks. Experiments show that scaling diffusion language models consistently improves performance across downstream language tasks. We further discover that instruction finetuning can elicit zero-shot and few-shot in-context learning abilities that help tackle many unseen tasks by following natural language instructions, and show promise in advanced and challenging abilities such as reasoning.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 90 of its 133 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

90 of 133 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Likelihood-Based Diffusion Language Models30 May
  2. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation16 May
  3. 2023A Cheaper and Better Diffusion Language Model with Soft-Masked Noise10 Apr
  4. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  5. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  6. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  7. 2022Continuous diffusion for categorical data28 Nov
  8. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  9. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  10. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct

Other references · 80

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  12. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
38 more
  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  13. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  14. 2023Let's Verify Step by Stepcited by 3,789
  15. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  16. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  17. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  18. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  19. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  20. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  21. 2022Emergent Abilities of Large Language Modelscited by 2,512
  22. 2018Deep contextualized word representationscited by 2,290
  23. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  24. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  25. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  26. 2020GLU Variants Improve Transformercited by 1,804
  27. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  28. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  29. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  30. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  31. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  32. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  33. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  34. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  35. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  36. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  37. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
  38. 2022Galactica: A Large Language Model for Sciencecited by 744

50 most cited of 80 All 80 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 22; 2026: 10.

39 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 912
  2. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  3. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  4. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 274
  5. 2024Scaling up Masked Diffusion Models on Textcited by 199
  6. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
  7. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  8. 2024Diffusion Language Models Are Versatile Protein Learnerscited by 98
  9. 2025A Survey on Diffusion Language Modelscited by 78
  10. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  11. 2025Theoretical Benefit and Limitation of Diffusion Language Modelcited by 47
  12. 2025ParaThinker: Native Parallel Thinking as a New Paradigm to Scale LLM Test-time Computecited by 38
27 more
  1. 2025TESS 2: A Large-Scale Generalist Diffusion Language Modelcited by 30
  2. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  3. 2025LaDiR: Latent Diffusion Enhances LLMs for Text Reasoningcited by 29
  4. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  5. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Modelscited by 23
  6. 2023Fast Sampling via Discrete Non-Markov Diffusion Models with Predetermined Transition Timecited by 20
  7. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Modelscited by 17
  8. 2025Elucidating the Design Space of Multimodal Protein Language Modelscited by 11
  9. 2025Toward Safer Diffusion Language Models: Discovery and Mitigation of Priming Vulnerabilitycited by 10
  10. 2024Transfer Learning for Text Diffusion Modelscited by 6
  11. 2026Context Tokens are Anchors: Understanding the Repetition Curse in dMLLMs from an Information Flow Perspectivecited by 6
  12. 2024Guided Discrete Diffusion for Electronic Health Record Generationcited by 5
  13. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  14. 2024Latent Diffusion Models for Controllable RNA Sequence Generationcited by 4
  15. 2025DMark: Order-Agnostic Watermarking for Diffusion Large Language Modelscited by 4
  16. 2025Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoningcited by 3
  17. 2025DiffuGR: Generative Document Retrieval with Diffusion Language Modelscited by 3
  18. 2025A Comprehensive Study on Visual Token Redundancy for Discrete Diffusion-based Multimodal Large Language Modelscited by 3
  19. 2026Towards A Generative Protein Evolution Machine with DPLM-Evocited by 3
  20. 2025Generation Properties of Stochastic Interpolation under Finite Training Setcited by 1
  21. 2026Measuring Temporal Linguistic Emergence in Diffusion Language Modelscited by 1
  22. 2026Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attentioncited by 1
  23. 2026Understanding Parallel Samplers in Masked Diffusion via Random Walks on Graphscited by 1
  24. 2026Sentence Curve Language Modelscited by 0
  25. 2026AR-MAP: Are Autoregressive Large Language Models Implicit Teachers for Diffusion Large Language Models?cited by 0
  26. 2026Efficient Self-Evaluation for Diffusion Language Models via Sequence Regenerationcited by 0
  27. 2026Read the Trace, Steer the Path: Trajectory-Aware Reinforcement Learning for Diffusion Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
  2. 2026Understanding and Accelerating the Training of Masked Diffusion Language Models
  3. 2026Improved Large Language Diffusion Models
  4. 2022Latent Diffusion for Language Generation
  5. 2025Large Language Models to Diffusion Finetuning
  6. 2026Why Instruction-Based Unlearning Fails in Diffusion Models?
  7. 2026Scaling Beyond Masked Diffusion Language Models
  8. 2026Mask-Aware Policy Gradients for Diffusion Language Models