mlsanityData to 7 Oct 2026

Variational Speculative Decoding: Rethinking Draft Training from Token Likelihood to Sequence Acceptance

arXiv 2602.05774 ↗First posted 5 Feb 2026cs.LG cs.AI math.PRcited by 2

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Speculative decoding accelerates inference for (M)LLMs, yet a training-decoding discrepancy persists: while existing methods optimize single greedy trajectories, decoding involves verifying and ranking multiple sampled draft paths. We propose Variational Speculative Decoding (VSD), formulating draft training as variational inference over latent proposals (draft paths). VSD maximizes the marginal probability of target-model acceptance, yielding an ELBO that promotes high-quality latent proposals while minimizing divergence from the target distribution. To enhance quality and reduce variance, we incorporate a path-level utility and optimize via an Expectation-Maximization procedure. The E-step draws Monte Carlo samples from an oracle-filtered posterior, while the M-step maximizes weighted likelihood using Adaptive Rejection Weighting (ARW) and Confidence-Aware Regularization (CAR). Theoretical analysis confirms that VSD increases expected acceptance length and speedup. Extensive experiments across LLMs and MLLMs show that VSD achieves up to a 9.6% speedup over EAGLE-3 and 7.9% over ViSpec, significantly improving decoding efficiency.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 32 of its 59 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

32 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  10. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  11. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  12. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
19 more
  1. 2023Speculative Decoding with Big Little Decodercited by 163
  2. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillationcited by 160
  3. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  4. 2024Hydra: Sequentially-Dependent Draft Heads for Medusa Decodingcited by 113
  5. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  6. 2024Learning Harmonized Representations for Speculative Samplingcited by 69
  7. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Accelerationcited by 69
  8. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  9. 2024LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decodingcited by 46
  10. 2024Ouroboros: Generating Longer Drafts Phrase by Phrase for Faster Speculative Decodingcited by 25
  11. 2025ViSpec: Accelerating Vision-Language Models with Vision-Aware Speculative Decodingcited by 21
  12. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decodingcited by 15
  13. 2025DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decodingcited by 13
  14. 2025Speculative Decoding Reimagined for Multimodal Large Language Modelscited by 10
  15. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decodingcited by 9
  16. 2025Speculative Jacobi-Denoising Decoding for Accelerating Autoregressive Text-to-image Generationcited by 7
  17. 2025SpecVLM: Fast Speculative Decoding in Vision-Language Modelscited by 4
  18. 2025HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Modelscited by 4
  19. 2025HPS: Hard Preference Sampling for Human Preference Alignmentcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Spec-AUF: Accept-Until-Fail Training under Train-Inference Misalignment for Masked Block Drafterscited by 3
  2. 2026APEX: Speculate smarter, not deepercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
  2. 2026Acceptance-Aware Draft Model Training for Speculative Decoding
  3. 2026A Practical Investigation of Training-free Relaxed Speculative Decoding
  4. 2026TAPS: Task Aware Proposal Distributions for Speculative Sampling
  5. 2026Faster LLM Inference via Sequential Monte Carlo
  6. 2026Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding
  7. 2025Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding
  8. 2025READER: Retrieval-Assisted Drafter for Efficient LLM Inference