mlsanityData to 7 Oct 2026

DistillSpec: Improving Speculative Decoding via Knowledge Distillation

arXiv 2310.08461 ↗First posted 12 Oct 2023cs.CL cs.AI cs.LGcited by 160

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) accelerates large language model inference by employing a faster draft model for generating multiple tokens, which are then verified in parallel by the larger target model, resulting in the text generated according to the target model distribution. However, identifying a compact draft model that is well-aligned with the target model is challenging. To tackle this issue, we propose DistillSpec that uses knowledge distillation to better align the draft model with the target model, before applying SD. DistillSpec makes two key design choices, which we demonstrate via systematic study to be crucial to improving the draft and target alignment: utilizing on-policy data generation from the draft model, and tailoring the divergence function to the task and decoding strategy. Notably, DistillSpec yields impressive 10 - 45% speedups over standard SD on a range of standard benchmarks, using both greedy and non-greedy sampling. Furthermore, we combine DistillSpec with lossy SD to achieve fine-grained control over the latency vs. task performance trade-off. Finally, in practical scenarios with models of varying sizes, first using distillation to boost the performance of the target model and then applying DistillSpec to train a well-aligned draft model can reduce decoding latency by 6-10x with minimal performance drop, compared to standard decoding without distillation.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 39 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

30 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Online Speculative Decoding11 Oct
  2. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  4. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  9. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2015Teaching Machines to Read and Comprehendcited by 1,039
14 more
  1. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  3. 2016Sequence-Level Knowledge Distillationcited by 761
  4. 2022Efficiently Scaling Transformer Inferencecited by 529
  5. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  6. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  7. 2023Large Language Models as Tool Makerscited by 293
  8. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  9. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  10. 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 112
  11. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 76
  12. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 68
  13. 2022Lossless Acceleration for Seq2seq Generation with Aggressive Decodingcited by 18
  14. 2021When in Doubt, Summon the Titans: Efficient Inference with Large Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 57; 2025: 52; 2026: 49.

160 papers.

Sort:

By area:

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  5. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  6. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  7. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  9. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
  10. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  11. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  12. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Lengthcited by 83
38 more
  1. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  2. 2025Universal Model Routing for Efficient LLM Inferencecited by 80
  3. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decodingcited by 74
  4. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  5. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  6. 2024CLLMs: Consistency Large Language Modelscited by 71
  7. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  8. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  9. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  10. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  11. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decodingcited by 46
  12. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  13. 2024Faster Cascades via Speculative Decodingcited by 38
  14. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Modelscited by 37
  15. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  16. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 36
  17. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 35
  18. 2024Multi-Candidate Speculative Decodingcited by 30
  19. 2024Block Verification Accelerates Speculative Decodingcited by 30
  20. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 28
  21. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacementcited by 27
  22. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  23. 2025SpecDiff-2: Scaling Diffusion Drafter Alignment For Faster Speculative Decodingcited by 26
  24. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  25. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  26. 2024Direct Alignment of Draft Model for Speculative Decoding with Chat-Fine-Tuned LLMscited by 21
  27. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  28. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  29. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  30. 2025Towards Optimal Multi-draft Speculative Decodingcited by 19
  31. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 18
  32. 2024Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Modelscited by 18
  33. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 18
  34. 2024Mixture of Attentions For Speculative Decodingcited by 17
  35. 2025SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMscited by 17
  36. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decodingcited by 17
  37. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 16
  38. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15

50 most cited of 160 All 160 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
  2. 2025Speculative Decoding: Performance or Illusion?
  3. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding
  4. 2024Temperature-Centric Investigation of Speculative Decoding with Knowledge Distillation
  5. 2024Boosting Lossless Speculative Decoding via Feature Sampling and Partial Alignment Distillation
  6. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  7. 2026Speculative Decoding and the Curse of Multilinguality
  8. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding