mlsanityData to 14 Oct 2026

Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding

arXiv 2504.00030 ↗First posted 28 Mar 2025cs.CL cs.AI cs.LGcited by 6

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding accelerates large language model (LLM) inference by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, selecting an optimal speculation length is critical for maximizing speedup while minimizing wasted computation. We introduce textitGammaTune and textitGammaTune+, training-free adaptive algorithms that dynamically adjust speculation length based on token acceptance rates using a heuristic-based switching mechanism. Evaluated on SpecBench across multiple tasks and model pairs, our method outperforms other heuristic-based approaches and fixed-length speculative decoding, achieving an average speedup of 15% (±5%) with textitGammaTune and 16% (±3%) with textitGammaTune+, while reducing performance variance. This makes textitGammaTune a robust and efficient solution for real-world deployment.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 17 of its 23 references that are in the map. First posted on arXiv 28 Mar 2025.

Builds on

17 of 23 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure15 Oct
  2. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  3. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  4. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement21 Feb
  5. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models16 Feb
  6. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  7. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  8. 2023Speculative Decoding with Big Little Decoder15 Feb
  9. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  10. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 7

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  5. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 79
  6. 2023ESPN: Memory-Efficient Multi-Vector Information Retrievalcited by 8
  7. 2024Are LLMs Good Annotators for Discourse-level Event Relation Extraction?cited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsitycited by 6
  2. 2025POSS: Position Specialist Generates Better Draft for Speculative Decodingcited by 2
  3. 2026PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Lengthcited by 2
  4. 2026The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Expertscited by 1
  5. 2026Trajectory-Retrieval Speculative Decoding: When Does a Model's Own History Help?cited by 0
  6. 2026SP2ec: Adaptive Self-Speculative Decoding for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
  2. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  3. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving
  4. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  5. 2025Speculative Decoding: Performance or Illusion?
  6. 2026LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimization
  7. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
  8. 2026Faster LLM Inference via Sequential Monte Carlo