mlsanityData to 14 Oct 2026

LogitSpec: Accelerating Retrieval-based Speculative Decoding via Next Next Token Speculation

arXiv 2507.01449 ↗First posted 2 Jul 2025cs.CLcited by 14

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD), where a small draft model is employed to propose draft tokens in advance and then the target model validates them in parallel, has emerged as a promising technique for LLM inference acceleration. Many endeavors to improve SD are to eliminate the need for a draft model and generate draft tokens in a retrieval-based manner in order to further alleviate the drafting overhead and significantly reduce the difficulty in deployment and applications. However, retrieval-based SD relies on a matching paradigm to retrieval the most relevant reference as the draft tokens, where these methods often fail to find matched and accurate draft tokens. To address this challenge, we propose LogitSpec to effectively expand the retrieval range and find the most relevant reference as drafts. Our LogitSpec is motivated by the observation that the logit of the last token can not only predict the next token, but also speculate the next next token. Specifically, LogitSpec generates draft tokens in two steps: (1) utilizing the last logit to speculate the next next token; (2) retrieving relevant reference for both the next token and the next next token. LogitSpec is training-free and plug-and-play, which can be easily integrated into existing LLM inference frameworks. Extensive experiments on a wide range of text generation benchmarks demonstrate that LogitSpec can achieve up to 2.61 × speedup and 3.28 mean accepted tokens per decoding step. Our code is available at https://github.com/smart-lty/LogitSpec.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 52 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

38 of 52 references.

Closest ancestors · 26

Same direction, newest first

  1. 2025Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding13 Mar
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  3. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verification24 Feb
  4. 2025CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Drafter24 Feb
  5. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment31 Jan
  6. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures25 Dec
  7. 2024SAM Decoding: Speculative Decoding via Suffix Automaton16 Nov
  8. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  9. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  10. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  11. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure25 Jun
  12. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
14 more
  1. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism6 Jun
  2. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths30 May
  3. 2024Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge1 May
  4. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  5. 2024GliDe with a CaPE: A Low-Hassle Method to Accelerate Speculative Decoding3 Feb
  6. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  7. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  9. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  10. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  11. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding9 Oct
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  13. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  14. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 12

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2024A Survey on Large Language Models for Code Generationcited by 658
  8. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  9. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  10. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  11. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  12. 2024Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recyclingcited by 47

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 13.

14 papers.

Sort:

By area:

  1. 2026TALON: Confidence-Aware Speculative Decoding with Adaptive Token Treescited by 13
  2. 2025Long-Context Modeling with Dynamic Hierarchical Sparse Attention for Memory-Constrained LLM Inferencecited by 7
  3. 2026Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decodingcited by 6
  4. 2026When Hidden States Drift: Can KV Caches Rescue Long-Range Speculative Decoding?cited by 5
  5. 2026D-cut: Adaptive Verification Depth Pruning for Batched Speculative Decodingcited by 5
  6. 2026RACER: Retrieval-Augmented Contextual Rapid Speculative Decodingcited by 4
  7. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decodingcited by 4
  8. 2026Bastion: Budget-Aware Speculative Decoding with Tree-structured Block Diffusion Draftingcited by 2
  9. 2026TailSieve: Partial-Rollout-Guided Tail Routing for LLM Rolloutscited by 2
  10. 2026SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decodingcited by 1
  11. 2026AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decodingcited by 1
  12. 2026Oilbird: Training-Free Speculative Decoding with Keys the Verifier Already Computescited by 0
2 more
  1. 2026ECHO: Early-layer Collaborative Hierarchical Orchestration with Bonus Logits in Speculative Decodingcited by 0
  2. 2026AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fillcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference
  2. 2025CARD: A Cache-Assisted Parallel Speculative Decoding Framework via Query-and-Correct Paradigm for Accelerating LLM Inference
  3. 2025Speculative Decoding: Performance or Illusion?
  4. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
  5. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  6. 2025HiSpec: Hierarchical Speculative Decoding for LLMs
  7. 2025FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference
  8. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models