mlsanityData to 7 Oct 2026

Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies

arXiv 2502.05202 ↗First posted 31 Jan 2025cs.CL cs.AI cs.LGcited by 20

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Accelerating the inference of large language models (LLMs) is a critical challenge in generative AI. Speculative decoding (SD) methods offer substantial efficiency gains by generating multiple tokens using a single target forward pass. However, existing SD approaches require the drafter and target models to share the same vocabulary, thus limiting the pool of possible drafters, often necessitating the training of a drafter from scratch. We present three new SD methods that remove this shared-vocabulary constraint. All three methods preserve the target distribution (i.e., they are lossless) and work with off-the-shelf models without requiring additional training or modifications. Empirically, on summarization, programming, and long-context tasks, our algorithms demonstrate significant speedups of up to 2.8x over standard autoregressive decoding. By enabling any off-the-shelf model to serve as a drafter and requiring no retraining, this work substantially broadens the applicability of the SD framework in practice.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 15 of its 34 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

15 of 34 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding20 Aug
  2. 2024Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference23 May
  3. 2024Dynamic Speculation Lookahead Accelerates Speculative Decoding of Large Language Models7 May
  4. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
  5. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  6. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 9

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  8. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  9. 2022Efficiently Scaling Transformer Inferencecited by 529

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 13.

20 papers.

Sort:

By area:

  1. 2025DiffuSpec: Unlocking Diffusion Language Models for Speculative Decodingcited by 40
  2. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 4
  3. 2025OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decodingcited by 4
  4. 2025DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Modelscited by 4
  5. 2026Agent-X: Full Pipeline Acceleration of On-device AI Agentscited by 4
  6. 2025When, What, and How: Rethinking Retrieval-Enhanced Speculative Decodingcited by 3
  7. 2026ConfLayers: Adaptive Confidence-based Layer Skipping for Self-Speculative Decodingcited by 3
  8. 2025Out-of-Vocabulary Sampling Boosts Speculative Decodingcited by 2
  9. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
  10. 2026SpecHop: Continuous Speculation for Accelerating Multi-Hop Retrieval Agentscited by 1
  11. 2026NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabulariescited by 1
  12. 2026AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMscited by 1
8 more
  1. 2025TokenTiming: A Dynamic Alignment Method for Universal Speculative Decoding Model Pairscited by 0
  2. 2026Task-Centric Acceleration of Small-Language Modelscited by 0
  3. 2026Cross-Family Speculative Decoding for Polish Language Models on Apple~Silicon: An Empirical Evaluation of Bielik~11B with UAG-Extended MLX-LMcited by 0
  4. 2026SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selectioncited by 0
  5. 2026Mitigating Hallucinations in Large Language Models Via Decoder Layer Skippingcited by 0
  6. 2026Diffusion Language Model Parallel Decoding via Product-of-Experts Bridgecited by 0
  7. 2026X-CoSD: Communication-Efficient Cross-Vocabulary Collaborative Speculative Decodingcited by 0
  8. 2026Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference
  2. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  3. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding
  4. 2024Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
  5. 2024Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution
  6. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  7. 2024Speculative Streaming: Fast LLM Inference without Auxiliary Models
  8. 2024Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism