mlsanityData to 14 Oct 2026

EvoSpec: Evolving Speculative Decoding via Real-Time Vocabulary and Parameter Adaptation

arXiv 2605.27390 ↗First posted 17 Apr 2026cs.CL cs.AIcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding accelerates Large Language Model inference through draft-then-verify generation, yet lightweight draft models face coupled efficiency and quality limitations: large-vocabulary output projection is costly, while limited draft capacity and static parameters reduce acceptance under specialized or shifting inputs. Vocabulary pruning lowers projection cost, but static variants miss locally important long-tail tokens, while dynamic variants remain sensitive to preset selection policies and budgets. Moreover, limited draft capacity can leave the draft distribution misaligned even when the target token is covered. Online alignment improves draft quality, but full-parameter updates introduce substantial memory and latency overhead. We introduce EvoSpec, which jointly adapts the active vocabulary and lightweight draft parameters from verification feedback. EvoSpec asynchronously retrieves semantic and statistical token neighbors and performs curriculum-weighted online LoRA alignment while preserving exact target-model verification. On Qwen3-8B/EAGLE-2, EvoSpec reaches a 2.18× speedup over vanilla decoding and a 1.20× gain over EAGLE-2, while improving specialized-domain coverage and using 27% less auxiliary GPU adaptation memory than full-parameter online adaptation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 38 of its 46 references that are in the map. First posted on arXiv 17 Apr 2026.

Builds on

38 of 46 references.

Closest ancestors · 18

Same direction, newest first

  1. 2026NanoSpec: Accelerating Speculative Decoding using Minimalist In-Context Vocabularies8 Apr
  2. 2026Speculative Decoding with a Speculative Vocabulary14 Feb
  3. 2025FastMTP: Accelerating LLM Inference with Enhanced Multi-Token Prediction16 Sep
  4. 2025VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs28 Jun
  5. 2025Training Domain Draft Models for Speculative Decoding: Best Practices and Insights10 Mar
  6. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  7. 2025FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling20 Feb
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  9. 2024Recurrent Drafter for Fast Speculative Decoding in Large Language Models14 Mar
  10. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decoding3 Feb
  11. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
6 more
  1. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding15 Jan
  2. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  3. 2023Online Speculative Decoding11 Oct
  4. 2023Speculative Decoding with Big Little Decoder15 Feb
  5. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  6. 2022Fast Inference from Transformers via Speculative Decoding30 Nov

Other references · 20

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  11. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  12. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
8 more
  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  3. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  4. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  5. 2024DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimizationcited by 5
  6. 2025MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Modelscited by 5
  7. 2024COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinementcited by 3
  8. 2025EfficientXpert: Efficient Domain Adaptation for Large Language Models via Propagation-Aware Pruningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Drafts Evolve: Speculative Decoding Meets Online Learning
  2. 2025AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference
  3. 2025Training-Free Loosely Speculative Decoding: Accepting Semantically Correct Drafts Beyond Exact Match
  4. 2025Steering Pretrained Drafters during Speculative Decoding
  5. 2024ParallelSpec: Parallel Drafter for Efficient Speculative Decoding
  6. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  7. 2024SSSD: Simply-Scalable Speculative Decoding
  8. 2026AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding