mlsanityData to 7 Oct 2026

Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

arXiv 2503.09573 ↗First posted 12 Mar 2025cs.LG cs.AIcited by 385

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lms

Key facts. Cited by 385 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 65 of its 74 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

65 of 74 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling4 Sep
  2. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  4. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  5. 2024Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models12 Feb
  6. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct
  7. 2023Likelihood-Based Diffusion Language Models30 May
  8. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation16 May
  9. 2022Continuous diffusion for categorical data28 Nov
  10. 2022DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models28 Nov
  11. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct
  12. 2021Autoregressive Diffusion Models5 Oct

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Scalable Diffusion Models with Transformerscited by 5,931
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  12. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
38 more
  1. 2016Pointer Sentinel Mixture Modelscited by 2,992
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  6. 2021Variational Diffusion Modelscited by 1,145
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  8. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  9. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  11. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  12. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
  13. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  14. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 566
  15. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  16. 2017Non-Autoregressive Neural Machine Translationcited by 457
  17. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 437
  18. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 362
  19. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  20. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  21. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  22. 2019Levenshtein Transformercited by 197
  23. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 180
  24. 2022Score-based Continuous-time Discrete Diffusion Modelscited by 167
  25. 2024Simple Guidance Mechanisms for Discrete Diffusion Modelscited by 148
  26. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  27. 2024Unlocking Guidance for Discrete State-Space Diffusion and Flow Modelscited by 124
  28. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  29. 2023Stay on topic with Classifier-Free Guidancecited by 90
  30. 2023Dirichlet Diffusion Score Model for Biological Sequence Generationcited by 73
  31. 2024CLLMs: Consistency Large Language Modelscited by 71
  32. 2013A Deep and Tractable Density Estimatorcited by 67
  33. 2023Bayesian Flow Networkscited by 62
  34. 2023Diffusion Models With Learned Adaptive Noisecited by 50
  35. 2023InfoDiffusion: Representation Learning Using Information Maximizing Diffusion Modelscited by 48
  36. 2024Pard: Permutation-Invariant Autoregressive Diffusion for Graph Generationcited by 25
  37. 2025Latent Thought Models with Variational Bayes Inference-Time Computationcited by 25
  38. 2021Autoregressive Quantile Flows for Predictive Uncertainty Estimationcited by 15

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 132; 2026: 253.

385 papers.

Sort:

By area:

  1. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  2. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  3. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 184
  4. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  6. 2025Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inferencecited by 173
  7. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  8. 2025dKV-Cache: The Cache for Diffusion Language Modelscited by 140
  9. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  10. 2025Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmaskingcited by 128
  11. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 128
  12. 2025The Diffusion Dualitycited by 124
38 more
  1. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  2. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  3. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 111
  4. 2025Accelerating Diffusion LLMs via Adaptive Parallel Decodingcited by 108
  5. 2025SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformercited by 91
  6. 2025LaViDa: A Large Diffusion Language Model for Multimodal Understandingcited by 81
  7. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  8. 2025A Survey on Diffusion Language Modelscited by 75
  9. 2025Diffusion Language Models Know the Answer Before Decodingcited by 68
  10. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  11. 2025Discrete Diffusion in Large Language and Multimodal Models: A Surveycited by 61
  12. 2025Diffusion Beats Autoregressive in Data-Constrained Settingscited by 61
  13. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 59
  14. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Modelcited by 54
  15. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Modelscited by 53
  16. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 48
  17. 2025Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMscited by 45
  18. 2025Diffusion Language Models are Super Data Learnerscited by 45
  19. 2025Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principlescited by 43
  20. 2025ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMscited by 42
  21. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 38
  22. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 38
  23. 2025Don't Settle Too Early: Self-Reflective Remasking for Diffusion Language Modelscited by 37
  24. 2025TiDAR: Think in Diffusion, Talk in Autoregressioncited by 35
  25. 2025Plan for Speed: Dilated Scheduling for Masked Diffusion Language Modelscited by 34
  26. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMscited by 33
  27. 2025Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Modelscited by 33
  28. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Modelscited by 32
  29. 2025Attention Is All You Need for KV Cache in Diffusion LLMscited by 32
  30. 2025CANDI: Hybrid Discrete-Continuous Diffusion Modelscited by 31
  31. 2025Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generationcited by 30
  32. 2025AdaBlock-dLLM: Semantic-Aware Diffusion LLM Inference via Adaptive Block Sizecited by 30
  33. 2025Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimizationcited by 30
  34. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 30
  35. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speedcited by 30
  36. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 29
  37. 2025Learning Unmasking Policies for Diffusion Language Modelscited by 29
  38. 2026LangFlow: Continuous Diffusion Rivals Discrete in Language Modelingcited by 28

50 most cited of 385 All 385 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Block Diffusion: Resolving Training-Inference Mismatch in Diffusion Language Models
  2. 2026Scaling Beyond Masked Diffusion Language Models
  3. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Models
  4. 2026Optimizers for Diffusion Models: A Controlled Benchmark
  5. 2025Optimal Inference Schedules for Masked Diffusion Models
  6. 2026Rethinking the Generation Order of Block Diffusion Language Models
  7. 2026LESS Is More: Mutual-Stability Sampling for Diffusion Language Models
  8. 2026Is Your Diffusion Sampler Actually Correct? A Sampler-Centric Evaluation of Discrete Diffusion Language Models