mlsanityData to 7 Oct 2026

Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

arXiv 2402.12374 ↗First posted 19 Feb 2024cs.CLcited by 103

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

As the usage of large language models (LLMs) grows, performing efficient inference with these models becomes increasingly important. While speculative decoding has recently emerged as a promising direction for speeding up inference, existing methods are limited in their ability to scale to larger speculation budgets, and adapt to different hyperparameters and hardware. This paper introduces Sequoia, a scalable, robust, and hardware-aware algorithm for speculative decoding. To attain better scalability, Sequoia introduces a dynamic programming algorithm to find the optimal tree structure for the speculated tokens. To achieve robust speculative performance, Sequoia uses a novel sampling and verification method that outperforms prior work across different decoding temperatures. Finally, Sequoia introduces a hardware-aware tree optimizer that maximizes speculative performance by automatically selecting the token tree size and depth for a given hardware platform. Evaluation shows that Sequoia improves the decoding speed of Llama2-7B, Llama2-13B, and Vicuna-33B on an A100 by up to 4.04×, 3.73×, and 2.27×. For offloading setting on L40, Sequoia achieves as low as 0.56 s/token for exact Llama2-70B inference latency, which is 9.96× on our optimized offloading system (5.6 s/token), 9.7× than DeepSpeed-Zero-Inference, 19.5× than Huggingface Accelerate.

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 42 of its 52 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

42 of 52 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  4. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  5. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  6. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  7. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  9. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
23 more
  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  3. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  4. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  5. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  11. 2018Rethinking the Value of Network Pruningcited by 595
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  13. 2022Efficiently Scaling Transformer Inferencecited by 529
  14. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  15. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  16. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  17. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 328
  18. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  19. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  20. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  21. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 150
  22. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  23. 2023LLMCad: Fast and Scalable On-device Large Language Model Inferencecited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 40; 2026: 40.

103 papers.

Sort:

By area:

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  3. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  4. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  5. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 98
  6. 2025FlashDLM: Accelerating Diffusion Language Model Inference via Efficient KV Caching and Guided Diffusioncited by 77
  7. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  8. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  9. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  10. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
  11. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  12. 2024TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodputcited by 36
38 more
  1. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 36
  2. 2024SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applicationscited by 34
  3. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  4. 2025Self Speculative Decoding for Diffusion Large Language Modelscited by 29
  5. 2025PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptationcited by 28
  6. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  7. 2025SpeCa: Accelerating Diffusion Transformers with Speculative Feature Cachingcited by 27
  8. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  9. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structurecited by 25
  10. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  11. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decodingcited by 20
  12. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 20
  13. 2025Towards Optimal Multi-draft Speculative Decodingcited by 19
  14. 2025Towards AI Search Paradigmcited by 19
  15. 2024QSpec: Speculative Decoding with Complementary Quantization Schemescited by 18
  16. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 18
  17. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structurescited by 17
  18. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 17
  19. 2024Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inferencecited by 16
  20. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15
  21. 2025Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoffcited by 14
  22. 2025AutoJudge: Judge Decoding Without Manual Annotationcited by 13
  23. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  24. 2025Speculative Decoding: Performance or Illusion?cited by 13
  25. 2026TALON: Confidence-Aware Speculative Decoding with Adaptive Token Treescited by 12
  26. 2024SpecHub: Provable Acceleration to Multi-Draft Speculative Decodingcited by 10
  27. 2025SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMscited by 10
  28. 2024Inference Optimization of Foundation Models on AI Acceleratorscited by 9
  29. 2025CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Draftercited by 9
  30. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decodingcited by 9
  31. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 9
  32. 2025Spec-LLaVA: Accelerating Vision-Language Models with Dynamic Tree-Based Speculative Decodingcited by 8
  33. 2025Efficient Parallel Samplers for Recurrent-Depth Models and Their Connection to Diffusion Language Modelscited by 7
  34. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodingcited by 7
  35. 2024S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUscited by 6
  36. 2024Constrained Decoding with Speculative Lookaheadscited by 6
  37. 2024Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inferencecited by 5
  38. 2025Collaborative Speculative Inference for Efficient LLM Inference Servingcited by 5

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoding Speculative Decoding
  2. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  3. 2025Efficient Speculative Decoding for Llama at Scale: Challenges and Solutions
  4. 2026DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
  5. 2024HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models
  6. 2025Inference-Cost-Aware Dynamic Tree Construction for Efficient Inference in Large Language Models
  7. 2025DuoDecoding: Hardware-aware Heterogeneous Speculative Decoding with Dynamic Multi-Sequence Drafting
  8. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation