mlsanityData to 7 Oct 2026

Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff

arXiv 2502.20704 ↗First posted 28 Feb 2025cs.AIcited by 14

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative Decoding (SD) enforces strict distributional equivalence to the target model when accepting candidate tokens. While it maintains the target model's generation quality, this strict equivalence limits the speedup achievable by SD and prevents users from trading deviations from the target distribution in exchange for further inference speed gains. To address these limitations, we introduce Fuzzy Speculative Decoding (FSD) - a decoding algorithm that generalizes SD by accepting candidate tokens based on the divergences between the target and draft model distributions. By allowing for controlled divergence from the target model, FSD enables users to flexibly trade generation quality for inference speed. Across several benchmarks, our method is able to achieve significant runtime improvements of over 5 tokens per second faster than SD at only an approximate 2% absolute reduction in benchmark accuracy. In many cases, FSD is even able to match SD benchmark accuracy at over 2 tokens per second faster, demonstrating that distributional equivalence is not necessary to maintain target model performance. Furthermore, FSD can be seamlessly integrated into existing SD extensions; we demonstrate this by applying FSD to EAGLE-2, greatly enhancing this existing extension's efficiency while allowing it to leverage FSD's tunable quality-speed trade-off.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 24 of its 28 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

24 of 28 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment31 Jan
  2. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  3. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  5. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism6 Jun
  6. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  7. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  8. 2024Tandem Transformers for Inference Efficient LLMs13 Feb
  9. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  10. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
5 more
  1. 2023Accelerating Transformer Inference for Translation via Parallel Decoding17 May
  2. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  3. 2023Speculative Decoding with Big Little Decoder15 Feb
  4. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  5. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 7

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  7. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2026See the Forest for the Trees: Loosely Speculative Decoding via Visual-Semantic Guidance for Efficient Inference of Video LLMscited by 5
  2. 2025Accelerating Mobile Language Model via Speculative Decoding and NPU-Coordinated Executioncited by 4
  3. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inferencecited by 2
  4. 2026A Practical Investigation of Training-free Relaxed Speculative Decodingcited by 2
  5. 2026When Is a Draft Accepted? A Theory of Acceptance in Speculative Decodingcited by 1
  6. 20253-Model Speculative Decodingcited by 0
  7. 2026Faster LLM Inference via Sequential Monte Carlocited by 0
  8. 2026Beyond the Target: From Imitation to Collaboration in Speculative Decodingcited by 0
  9. 2026Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modescited by 0
  10. 2026Approximate Speculative Decodingcited by 0
  11. 2026UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decodingcited by 0
  12. 2026Trajectory-Retrieval Speculative Decoding: When Does a Model's Own History Help?cited by 0
2 more
  1. 2026Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distributioncited by 0
  2. 2026Secure Speculative Decoding for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Speculative Speculative Decoding
  2. 2025Entropy-Aware Token Rejection for Improving Speculative Decoding
  3. 2024SSSD: Simply-Scalable Speculative Decoding
  4. 2025Speculative Decoding: Performance or Illusion?
  5. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  6. 2026Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
  7. 2024Decoding Speculative Decoding
  8. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding