mlsanityData to 14 Oct 2026

Beyond Autoregression: Fast LLMs via Self-Distillation Through Time

arXiv 2410.21035 ↗First posted 28 Oct 2024cs.LG cs.CLcited by 78

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Autoregressive (AR) Large Language Models (LLMs) have demonstrated significant success across numerous tasks. However, the AR modeling paradigm presents certain limitations; for instance, contemporary autoregressive LLMs are trained to generate one token at a time, which can result in noticeable latency. Recent advances have indicated that search and repeated sampling can enhance performance in various applications, such as theorem proving, code generation, and alignment, by utilizing greater computational resources during inference. In this study, we demonstrate that diffusion language models are capable of generating at least 32 tokens simultaneously, while exceeding the performance of AR models in text quality and on the LAMBADA natural language understanding benchmark. This outcome is achieved through a novel distillation method for discrete diffusion models, which reduces the number of inference steps by a factor of 32-64. Practically, at the 1.3B parameters scale, diffusion models, even without caching, can generate tokens at a rate that is up to 8 times faster than AR models employing KV-caching, and we anticipate further improvements with the inclusion of caching. Moreover, we demonstrate the efficacy of our approach for diffusion language models with up to 860M parameters.

Key facts. Cited by 78 papers in the map (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 53 of its 89 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

53 of 89 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Promises, Outlooks and Challenges of Diffusion Language Modeling17 Jun
  2. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  4. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Data6 Jun
  5. 2023Likelihood-Based Diffusion Language Models30 May
  6. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control31 Oct

Other references · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  9. 2020Denoising Diffusion Implicit Modelscited by 9,781
  10. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  12. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
35 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  3. 2022Scalable Diffusion Models with Transformerscited by 6,054
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  13. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  14. 2023Consistency Modelscited by 1,822
  15. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  16. 2021Variational Diffusion Modelscited by 1,153
  17. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  18. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  19. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  20. 2021Argmax Flows and Multinomial Diffusion: Learning Categorical Distributionscited by 574
  21. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  22. 2022A Continuous Time Framework for Discrete Denoising Modelscited by 451
  23. 2022Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioningcited by 373
  24. 2021Knowledge Distillation in Iterative Generative Models for Improved Sampling Speedcited by 348
  25. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
  26. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  27. 2023Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Modelscited by 236
  28. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  29. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  30. 2020Combining Deep Reinforcement Learning and Search for Imperfect-Information Gamescited by 114
  31. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
  32. 2021Scaling Scaling Laws with Board Gamescited by 63
  33. 2023A Comprehensive Survey on Knowledge Distillation of Diffusion Modelscited by 45
  34. 2019Improving Policies via Search in Cooperative Partially Observable Gamescited by 33
  35. 2024Unified Discrete Diffusion for Categorical Datacited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 28; 2026: 50.

78 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 912
  2. 2025The Diffusion Dualitycited by 135
  3. 2025A Survey on Diffusion Language Modelscited by 78
  4. 2025Diffusion Language Models Know the Answer Before Decodingcited by 70
  5. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  6. 2025KLASS: KL-Guided Fast Inference in Masked Diffusion Modelscited by 47
  7. 2026ELF: Embedded Language Flowscited by 45
  8. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 37
  9. 2025Continuously Augmented Discrete Diffusion model for Categorical Generative Modelingcited by 34
  10. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  11. 2026Discrete Flow Mapscited by 29
  12. 2025Breaking AR's Sampling Bottleneck: Provable Acceleration via Diffusion Language Modelscited by 23
38 more
  1. 2025MDNS: Masked Diffusion Neural Sampler via Stochastic Optimal Controlcited by 21
  2. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21
  3. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 21
  4. 2025Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Maskingcited by 20
  5. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  6. 2025FS-DFM: Fast and Accurate Long Text Generation with Few-Step Diffusion Language Modelscited by 17
  7. 2025Ultra-Fast Language Generation via Discrete Diffusion Divergence Instructcited by 16
  8. 2025Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decodingcited by 15
  9. 2025ReDi: Rectified Discrete Flowcited by 14
  10. 2026Self-conditioned Flow Map Language Models via Fixed-point Flowscited by 14
  11. 2026The Diffusion Duality, Chapter II: Psi-Samplerscited by 12
  12. 2026Beyond Single Tokens: Distilling Discrete Diffusion Models via Discrete MMDcited by 12
  13. 2025Self-Speculative Masked Diffusionscited by 11
  14. 2026Posterior Refinement: Fast Language Generation via Any-Order Flow Mapscited by 11
  15. 2025Demystifying MaskGIT Sampler and Beyond: Adaptive Order Selection in Masked Diffusioncited by 10
  16. 2026Residual Context Diffusion Language Modelscited by 10
  17. 2026DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmaskingcited by 9
  18. 2025Learnable Sampler Distillation for Discrete Diffusion Modelscited by 8
  19. 2026IDLM: Inverse-distilled Diffusion Language Modelscited by 8
  20. 2026Introspective Diffusion Language Modelscited by 8
  21. 2026Scaling Categorical Flow Mapscited by 8
  22. 2025Partition Generative Modeling: Masked Modeling Without Maskscited by 6
  23. 2025Soft-Di[M]O: Improving One-Step Discrete Image Generation with Soft Embeddingscited by 6
  24. 2025Dimathtt[M]O: Distilling Masked Diffusion Models into One-step Generatorcited by 5
  25. 2025PairFlow: Closed-Form Source-Target Coupling for Few-Step Generation in Discrete Flow Modelscited by 5
  26. 2026Fixed-Point Masked Generative Modelingcited by 5
  27. 2026ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusioncited by 4
  28. 2025Simple Denoising Diffusion Language Modelscited by 2
  29. 2025Don't Let It Fade: Preserving Edits in Diffusion Language Models via Token Timestep Allocationcited by 2
  30. 2026A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMscited by 2
  31. 2026Coupling Models for One-Step Discrete Generationcited by 2
  32. 2026Drifting Objectives for Refining Discrete Diffusion Language Modelscited by 2
  33. 2026Masked Language Flow Modelscited by 2
  34. 2026Distilled Continuous Diffusion Language Models Can Write Code in Few Steps---or Onecited by 2
  35. 2025Accelerating Inference of Masked Image Generators via Reinforcement Learningcited by 1
  36. 2026VocalNet-MDM: Accelerating Streaming Speech LLM via Self-Distilled Masked Diffusion Modelingcited by 1
  37. 2026LADR: Locality-Aware Dynamic Rescue for Efficient Text-to-Image Generation with Diffusion Large Language Modelscited by 1
  38. 2026Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inferencecited by 1

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Unlocking Lossless Speedups in LLMs via Discrete Diffusion
  2. 2026Energy Consumption Analysis of Discrete Diffusion and Autoregressive Language Models
  3. 2025Beyond Next-Token Prediction: A Performance Characterization of Diffusion versus Autoregressive Language Models
  4. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
  5. 2025On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond
  6. 2025Dream 7B: Diffusion Large Language Models
  7. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models
  8. 2026Diffusion Language Models: An Experimental Analysis