mlsanityData to 7 Oct 2026

Large Language Diffusion Models

arXiv 2502.09992 ↗First posted 14 Feb 2025cs.CL cs.LGcited by 872

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: https://ml-gsai.github.io/LLaDA-demo/.

Key facts. Cited by 872 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 97 of its 122 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

97 of 122 references.

Closest ancestors · 22

Same direction, newest first

  1. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
  2. 2024Scaling up Masked Diffusion Models on Text24 Oct
  3. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  4. 2024Think While You Generate: Discrete Diffusion with Planned Denoising8 Oct
  5. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  6. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  7. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  8. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  9. 2024Diffusion On Syntax Trees For Program Synthesis30 May
  10. 2023DiSK: A Diffusion Model for Structured Knowledge8 Dec
  11. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning23 Aug
  12. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model5 Jun
10 more
  1. 2023Likelihood-Based Diffusion Language Models30 May
  2. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  3. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
  4. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise22 Dec
  5. 2022Continuous diffusion for categorical data28 Nov
  6. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  7. 2022Self-conditioned Embedding Diffusion for Text Generation8 Nov
  8. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  9. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Models17 Oct
  10. 2021Autoregressive Diffusion Models5 Oct

Other references · 75

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  11. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,053
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2022Classifier-Free Diffusion Guidancecited by 5,717
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Program Synthesis with Large Language Modelscited by 3,509
  13. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  14. 2023Mistral 7Bcited by 3,065
  15. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  16. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  17. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  18. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  19. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  20. 2023A Survey of Large Language Modelscited by 2,537
  21. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  22. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  23. 2020GLU Variants Improve Transformercited by 1,724
  24. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  25. 2019Root Mean Square Layer Normalizationcited by 1,425
  26. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  27. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  28. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  29. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  30. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  31. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  32. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  33. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  34. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  35. 2024Qwen2-Audio Technical Reportcited by 543
  36. 2022All are Worth Words: A ViT Backbone for Diffusion Modelscited by 498
  37. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  38. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 305; 2026: 567.

872 papers.

Sort:

By area:

  1. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  2. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 404
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  4. 2025Memory in the Age of AI Agentscited by 285
  5. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 273
  6. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  7. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  8. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  9. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  10. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 168
  11. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 145
  12. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
38 more
  1. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  2. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  3. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  4. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 111
  5. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  6. 2025Discrete Diffusion VLA: Bringing Discrete Diffusion to Action Decoding in Vision-Language-Action Policiescited by 90
  7. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  8. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  9. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  10. 2025Dream-Coder 7B: An Open Diffusion Language Model for Codecited by 81
  11. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  12. 2025A Survey on Diffusion Language Modelscited by 75
  13. 2025Diffusion Language Models Know the Answer Before Decodingcited by 68
  14. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  15. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  16. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  17. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  18. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 58
  19. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  20. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 53
  21. 2025Edit Flows: Flow Matching with Edit Operationscited by 53
  22. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  23. 2025Any-Order Flexible Length Masked Diffusioncited by 48
  24. 2025FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocitiescited by 46
  25. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  26. 2025Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMscited by 45
  27. 2025Diffusion Language Models are Super Data Learnerscited by 45
  28. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Modelscited by 44
  29. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  30. 2025When To Solve, When To Verify: Compute-Optimal Problem Solving and Generative Verification for LLM Reasoningcited by 41
  31. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  32. 2025Fine-Tuning Masked Diffusion for Provable Self-Correctioncited by 39
  33. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 38
  34. 2025DPad: Efficient Diffusion Language Models with Suffix Dropoutcited by 38
  35. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 38
  36. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 38
  37. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  38. 2026Categorical Flow Mapscited by 37

50 most cited of 872 All 872 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B
  2. 2023David helps Goliath: Inference-Time Collaboration Between Small Specialized and Large General Diffusion LMs
  3. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
  4. 2026Recipes for Steering and Scaling LLMs via Sampling
  5. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  6. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  7. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Models
  8. 2026LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference