mlsanityData to 7 Oct 2026

SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

arXiv 2305.09781 ↗First posted 16 May 2023cs.CL cs.DC cs.LGcited by 359

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality. Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1.5-2.8x for distributed LLM inference and by 2.6-3.5x for offloading-based LLM inference, while preserving the same generative performance. SpecInfer is publicly available at https://github.com/flexflow/FlexFlow/

Key facts. Cited by 359 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 56 references that are in the map. First posted on arXiv 16 May 2023.

Builds on

28 of 56 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  2. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
14 more
  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  2. 2023Instruction Tuning with GPT-4cited by 676
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  6. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  7. 2021Ensemble deep learning: A reviewcited by 283
  8. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  9. 2020Ansor: Generating High-Performance Tensor Programs for Deep Learningcited by 199
  10. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  11. 2021Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Maskscited by 103
  12. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  13. 2019Pretraining-Based Natural Language Generation for Text Summarizationcited by 40
  14. 2020Tree-structured Attention with Hierarchical Accumulationcited by 30

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 69; 2025: 137; 2026: 149.

359 papers.

Sort:

By area:

  1. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  2. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 266
  3. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  4. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  5. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  6. 2023Efficient Large Language Models: A Surveycited by 183
  7. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  8. 2023Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generationcited by 103
  9. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  10. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  11. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  12. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
38 more
  1. 2024XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Modelscited by 73
  2. 2025DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMscited by 73
  3. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  4. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  5. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Servingcited by 68
  6. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  7. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 60
  8. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 59
  9. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  10. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 54
  11. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 48
  12. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  13. 2025Your LLM Knows the Future: Uncovering Its Multi-Token Prediction Potentialcited by 46
  14. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  15. 2024Faster Cascades via Speculative Decodingcited by 38
  16. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 38
  17. 2024SAM Decoding: Speculative Decoding via Suffix Automatoncited by 38
  18. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  19. 2025OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agentscited by 37
  20. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 35
  21. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34
  22. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  23. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  24. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacementcited by 27
  25. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  26. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  27. 2025SpeCa: Accelerating Diffusion Transformers with Speculative Feature Cachingcited by 27
  28. 2026Accelerating Speculative Decoding with Block Diffusion Draft Treescited by 27
  29. 2024Compute Or Load KV Cache? Why Not Both?cited by 25
  30. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structurecited by 25
  31. 2024A Survey: Collaborative Hardware and Software Design in the Era of Large Language Modelscited by 21
  32. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  33. 2025Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphscited by 21
  34. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  35. 2025Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabulariescited by 20
  36. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Samplingcited by 20
  37. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  38. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19

50 most cited of 359 All 359 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Hierarchical Verification of Speculative Beams for Accelerating LLM Inference
  2. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decoding
  3. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting
  4. 2024PipeInfer: Accelerating LLM Inference using Asynchronous Pipelined Speculation
  5. 2026SpecQuant: Speculative Decoding with Multi-Parent Quantization for Adaptive LLM Inference
  6. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  7. 2025Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models
  8. 2023Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding