mlsanityData to 7 Oct 2026

VIA-SD: Verification via Intra-Model Routing for Speculative Decoding

arXiv 2606.12243 ↗First posted 10 Jun 2026cs.CL cs.AIcited by 0

On the map: Efficient LLM inference and serving → Speculative decoding

Abstract (arXiv)

Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier via intra-model routing, instead of the full verifier. This motivates our slim-verifier to handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), a multi-tier framework using a routed slim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases, slim-verifier regeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10-0.22 and delivers 10-20% speedups over strong SD baselines, while achieving 2.5-3x acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Speculative decoding, one of 12 directions in Efficient LLM inference and serving. Builds on 37 of its 51 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

37 of 51 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025CLaSp: In-Context Layer Skip for Self-Speculative Decoding30 May
  2. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test3 Mar
  3. 2024SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration9 Oct
  4. 2024PEARL: Parallel Speculative Decoding with Adaptive Draft Length13 Aug
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees24 Jun
  6. 2024Faster Cascades via Speculative Decoding29 May
  7. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding19 Feb
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads19 Jan
  9. 2023Cascade Speculative Drafting for Even Faster LLM Inference18 Dec
  10. 2023DistillSpec: Improving Speculative Decoding via Knowledge Distillation12 Oct
  11. 2023Online Speculative Decoding11 Oct
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification16 May
4 more
  1. 2023Speculative Decoding with Big Little Decoder15 Feb
  2. 2023Accelerating Large Language Model Decoding with Speculative Sampling2 Feb
  3. 2022Fast Inference from Transformers via Speculative Decoding30 Nov
  4. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation30 Mar

Other references · 21

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2021Program Synthesis with Large Language Modelscited by 3,509
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
9 more
  1. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  2. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  3. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  4. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  5. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  6. 2025Dropping Experts, Recombining Neurons: Retraining-Free Pruning for Sparse Mixture-of-Experts LLMscited by 25
  7. 2024Multisize Dataset Condensationcited by 19
  8. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinementcited by 6
  9. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Datacited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding
  2. 2025HiSpec: Hierarchical Speculative Decoding for LLMs
  3. 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
  4. 2026From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning
  5. 2025Fail Fast, Win Big: Rethinking the Drafting Strategy in Speculative Decoding via Diffusion LLMs
  6. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decoding
  7. 2024Block Verification Accelerates Speculative Decoding
  8. 2025Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference