mlsanityData to 14 Oct 2026

Online Speculative Decoding

arXiv 2310.07177 ↗First posted 11 Oct 2023cs.AI cs.CL cs.LGcited by 113

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding is a pivotal technique to accelerate the inference of large language models (LLMs) by employing a smaller draft model to predict the target model's outputs. However, its efficacy can be limited due to the low predictive accuracy of the draft model, particularly when faced with diverse text inputs and a significant capability gap between the draft and target models. We introduce online speculative decoding to address this challenge. The main idea is to continuously update the (multiple) draft model(s) on observed user query data. Adapting to query distribution mitigates the shifts between the training distribution of the draft model and the query distribution, enabling the draft model to more accurately predict the target model's outputs. We develop a prototype of online speculative decoding based on knowledge distillation and evaluate it using both synthetic and real query data. The results show a substantial increase in the token acceptance rate by 0.1 to 0.65, bringing 1.42x to 2.17x latency reduction. Our code is available at https://github.com/LiuXiaoxuanPKU/OSD.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 33 references that are in the map. First posted on arXiv 11 Oct 2023.

Builds on

19 of 33 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
  2. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,139
  12. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
4 more
  1. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  2. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  3. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  4. 2022Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferabilitycited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 34; 2025: 40; 2026: 37.

113 papers.

Sort:

By area:

  1. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  3. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  4. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  6. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 165
  7. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  9. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  10. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 75
  11. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 72
  12. 2024MambaByte: Token-free Selective State Space Modelcited by 59
38 more
  1. 2024Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsificationcited by 59
  2. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  3. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  4. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Modelscited by 37
  5. 2024A Declarative System for Optimizing AI Workloadscited by 37
  6. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 37
  7. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 37
  8. 2024Decoding Speculative Decodingcited by 34
  9. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  10. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 30
  11. 2024A Theoretical Perspective for Speculative Decoding Algorithmcited by 29
  12. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 29
  13. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 23
  14. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 23
  15. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 22
  16. 2024ZipAR: Parallel Auto-regressive Image Generation through Spatial Localitycited by 21
  17. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
  18. 2024Online Cascade Learning for Efficient Inference over Streamscited by 20
  19. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 20
  20. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Modelscited by 19
  21. 2024Continuous Speculative Decoding for Autoregressive Image Generationcited by 19
  22. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  23. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 17
  24. 2024Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interfacecited by 16
  25. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 16
  26. 2025Speculative Decoding: Performance or Illusion?cited by 15
  27. 2026JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Draftingcited by 15
  28. 2024The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculationcited by 14
  29. 2024BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Modelscited by 13
  30. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  31. 2025Fast and Accurate Causal Parallel Decoding using Jacobi Forcingcited by 13
  32. 2025Dynamic Speculative Agent Planningcited by 12
  33. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 12
  34. 2024Optimized Multi-Token Joint Decoding with Auxiliary Model for LLM Inferencecited by 11
  35. 2024Efficient Inference for Large Language Model-based Generative Recommendationcited by 11
  36. 2026Draft-OPD: On-Policy Distillation for Speculative Draft Modelscited by 11
  37. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 10
  38. 2025Beat the long tail: Distribution-Aware Speculative Decoding for RL Trainingcited by 10

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  3. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  4. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  5. 2026Speculative Decoding with a Speculative Vocabulary
  6. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  7. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  8. 2024Graph-Structured Speculative Decoding