mlsanityData to 7 Oct 2026

Speculative Decoding with a Speculative Vocabulary

arXiv 2602.13836 ↗First posted 14 Feb 2026cs.CLcited by 6

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding has rapidly emerged as a leading approach for accelerating language model (LM) inference, as it offers substantial speedups while yielding identical outputs. This relies upon a small draft model, tasked with predicting the outputs of the target model. State-of-the-art speculative decoding methods use a draft model comprising a single decoder layer and output embedding matrix, with the latter dominating drafting time for the latest LMs. Recent work has sought to address this output distribution bottleneck by reducing the vocabulary of the draft model. While this can improve throughput, it compromises speculation effectiveness when the target token is out-of-vocabulary. In this paper, we argue for vocabulary speculation as an alternative to a reduced vocabulary. We propose SpecVocab, an efficient and effective method that selects a vocabulary subset per decoding step. Across a variety of tasks, we show that SpecVocab can achieve a higher acceptance length than state-of-the-art speculative decoding method, EAGLE-3. Notably, this yields up to an 8.1% increase in average throughput over EAGLE-3.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 45 references that are in the map. First posted on arXiv 14 Feb 2026.

Builds on

30 of 45 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs28 Jun
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  3. 2025DReSD: Dense Retrieval for Speculative Decoding21 Feb
  4. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling20 Feb
  5. 2024Learning Harmonized Representations for Speculative Sampling28 Aug
  6. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  7. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding7 Feb
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  11. 2023REST: Retrieval-Based Speculative Decoding14 Nov
  12. 2023Accelerating LLM Inference with Staged Speculative Decoding8 Aug
5 more
  1. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  2. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  4. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar
  5. 2021Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding9 Jun

Other references · 13

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  6. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  10. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 113
  11. 2025Over-Tokenized Transformer: Vocabulary is Generally Worth Scalingcited by 42
  12. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
1 more
  1. 2020Load What You Need: Smaller Versions of Multilingual BERTcited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decodingcited by 31
  2. 2026Trees from Marginals: Autoregressive drafting with factorized priorscited by 2
  3. 2026WhiFlash: Accelerating Speculative Decoding with Token-Level Cross-Paradigm Routingcited by 1
  4. 2026SlimSpec: Low-Rank Draft LM-Head for Accelerated Speculative Decodingcited by 0
  5. 2026EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptationcited by 0
  6. 2026Fully Rotation-Equivariant Spectral-Spatial Learning for Multispectral Object Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Decoding Speculative Decoding
  2. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  3. 2024A Theoretical Perspective for Speculative Decoding Algorithm
  4. 2025Speculative Decoding: Performance or Illusion?
  5. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding
  6. 2026Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
  7. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths
  8. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models