mlsanityData to 7 Oct 2026

Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding

arXiv 2605.00342 ↗First posted 1 May 2026cs.CLcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Tree-based speculative decoding accelerates autoregressive generation by verifying multiple draft candidates in parallel, but this advantage weakens for sparse Mixture-of-Experts (MoE) models. As the draft tree grows, different branches activate different experts, expanding the union of activated experts and substantially increasing target-side verification cost. We propose EVICT, a training-free, hyperparameter-free, and lossless adaptive verification method for MoE speculative decoding. EVICT makes every verified token count by truncating the draft tree before target verification and retaining only the cost-effective prefix. It leverages fine-grained drafter signals to estimate candidate benefit, combines them with offline-profiled verification cost, and remains highly compatible with the high-performance graph-based serving framework SGLang. Extensive experiments on diverse MoE backbones and benchmarks show that EVICT achieves up to 2.35x speedup over autoregressive decoding and an average 1.21x speedup over the state-of-the-art baseline EAGLE-3, while significantly reducing unnecessary expert activations during verification.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 35 of its 41 references that are in the map. First posted on arXiv 1 May 2026.

Builds on

35 of 41 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding17 Feb
  2. 2025UCCL-EP: Portable Expert-Parallel Communication22 Dec
  3. 2025GRACE-MoE: Grouping and Replication with Locality-Aware Routing for Efficient Distributed MoE Inference29 Sep
  4. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding29 Aug
  5. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts17 Jun
  6. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE26 May
  7. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models18 Dec
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov

Other references · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  10. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
15 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  3. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 502
  4. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  5. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  6. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 147
  8. 2026DFlash: Block Diffusion for Flash Speculative Decodingcited by 120
  9. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 64
  10. 2023Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracycited by 37
  11. 2025Every Activation Boosted: Scaling General Reasoner to 1 Trillion Open Language Foundationcited by 32
  12. 2024Dynamic Depth Decoding: Faster Speculative Decoding for LLMscited by 20
  13. 2026SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decodingcited by 15
  14. 2026TALON: Confidence-Aware Speculative Decoding with Adaptive Token Treescited by 12
  15. 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarioscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decodingcited by 1
  2. 2026LibraSpec: Dynamic Diffusion-Based Speculative Decoding via Marginal-Gain-Driven Optimizationcited by 0
  3. 2026The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
  2. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
  3. 2026Nucleus Speculative Decoding: Plausibility-Aware Verification Beyond Exact Distribution
  4. 2026Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
  5. 2026Approximate Speculative Decoding
  6. 2026SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decoding
  7. 2024Block Verification Accelerates Speculative Decoding
  8. 2026DLoop: Looped Speculative Decoding