mlsanityData to 7 Oct 2026

GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding

arXiv 2402.02082 ↗First posted 3 Feb 2024cs.CLcited by 74

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding is a relatively new decoding framework that leverages small and efficient draft models to reduce the latency of LLMs. In this study, we introduce GliDe and CaPE, two low-hassle modifications to vanilla speculative decoding to further improve the decoding speed of a frozen LLM. Specifically, GliDe is a modified draft model architecture that reuses the cached keys and values from the target LLM, while CaPE is a proposal expansion method that uses the draft model's confidence scores to help select additional candidate tokens for verification. Extensive experiments on different benchmarks demonstrate that our proposed GliDe draft model significantly reduces the expected decoding latency. Additional evaluation using walltime reveals that GliDe can accelerate Vicuna models up to 2.17x and further extend the improvement to 2.61x with CaPE. We will release our code, data, and the trained draft models.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 19 of its 29 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

19 of 29 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023SpecTr: Fast Speculative Decoding via Optimal Transport23 Oct
  2. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  3. 2023Speculative Decoding with Big Little Decoder15 Feb
  4. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  5. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 14

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  5. 2016Sequence-Level Knowledge Distillationcited by 761
  6. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  7. 2017Non-Autoregressive Neural Machine Translationcited by 457
  8. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  9. 2022Confident Adaptive Language Modelingcited by 297
  10. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  11. 2023Inference with Reference: Lossless Acceleration of Large Language Modelscited by 63
  12. 2021Order-Agnostic Cross Entropy for Non-Autoregressive Machine Translationcited by 57
2 more
  1. 2023Leveraging Large Language Models for Pre-trained Recommender Systemscited by 42
  2. 2023Speculative Contrastive Decodingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 38; 2026: 22.

74 papers.

Sort:

By area:

  1. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  3. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  4. 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
  5. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  6. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  7. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  8. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignmentcited by 59
  9. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 19
  10. 2025SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruningcited by 19
  11. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Servingcited by 18
  12. 2025BanditSpec: Adaptive Speculative Decoding via Bandit Algorithmscited by 18
38 more
  1. 2024Mixture of Attentions For Speculative Decodingcited by 17
  2. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
  3. 2026D-PACE: Dynamic Position-Aware Cross-Entropy for Parallel Speculative Draftingcited by 16
  4. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 15
  5. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15
  6. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 15
  7. 2024Reinforcement Retrieval Leveraging Fine-grained Feedback for Fact Checking News Claims with Black-Box LLMcited by 14
  8. 2025Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoffcited by 14
  9. 2025LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculationcited by 13
  10. 2026Speculative Speculative Decodingcited by 13
  11. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decodingcited by 12
  12. 2026TALON: Confidence-Aware Speculative Decoding with Adaptive Token Treescited by 12
  13. 2024Efficient Inference for Large Language Model-based Generative Recommendationcited by 11
  14. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Expertscited by 11
  15. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Matchcited by 11
  16. 2024Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledgecited by 10
  17. 2025CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Draftercited by 9
  18. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decodingcited by 9
  19. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decodingcited by 9
  20. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 9
  21. 2026DFlare: Scaling Up Draft Capacity for Block Diffusion Speculative Decodingcited by 9
  22. 2025DEL: Context-Aware Dynamic Exit Layer for Efficient Self-Speculative Decodingcited by 8
  23. 2025AdaDecode: Accelerating LLM Decoding with Adaptive Layer Parallelismcited by 7
  24. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 7
  25. 2026PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decodingcited by 7
  26. 2025RASD: Retrieval-Augmented Speculative Decodingcited by 6
  27. 2024Amphista: Bi-directional Multi-head Decoding for Accelerating LLM Inferencecited by 5
  28. 2024Clover-2: Accurate Inference for Regressive Lightweight Speculative Decodingcited by 5
  29. 2025Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMscited by 5
  30. 2026P-EAGLE: Parallel-Drafting EAGLE with Scalable Trainingcited by 5
  31. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5
  32. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Modelscited by 4
  33. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 4
  34. 2025Reject Only Critical Tokens: Pivot-Aware Speculative Decodingcited by 4
  35. 2025Steering Pretrained Drafters during Speculative Decodingcited by 4
  36. 2026Overcoming Joint Intractability with Lossless Hierarchical Speculative Decodingcited by 4
  37. 2026ConfSpec: Efficient Step-Level Speculative Reasoning via Confidence-Gated Verificationcited by 4
  38. 2026When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?cited by 4

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  2. 2024Decoding Speculative Decoding
  3. 2023Accelerating LLM Inference with Staged Speculative Decoding
  4. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  5. 2025Speculative Decoding: Performance or Illusion?
  6. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  7. 2026A Practical Investigation of Training-free Relaxed Speculative Decoding
  8. 2025From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing