mlsanityData to 14 Oct 2026

The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts

arXiv 2609.22156 ↗First posted 26 Aug 2026cs.LG stat.MLcited by 1

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Speculative decoding in Mixture-of-Experts (MoE) models faces the problem of unstable verification cost caused by input-dependent expert loading. To study the physics of this process, we formulate speculation-budget selection as an offline Stochastic Shortest Path (SSP) problem over reference sequences and build a diagnostic Oracle that uses counterfactual simulation to account for MoE verification cost. A detailed analysis of the Oracle's decisions on the Qwen3-Coder and EAGLE-3 pairing, in the space of marginal deltas (Delta Space), shows that rejected candidates form a strict linear boundary. This result demonstrates that a complex global optimization is locally governed by a necessary condition balancing marginal cost against expected progress (fracDelta mathbbE[Cost]Delta mathbbE[a]), providing a rigorous mathematical reference point for designing future adaptive online heuristics.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 27 of its 33 references that are in the map. First posted on arXiv 26 Aug 2026.

Builds on

27 of 33 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts14 Jul
  2. 2026Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding1 May
  3. 2026SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding11 Apr
  4. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding17 Feb
  5. 2025Accelerate Speculative Decoding with Sparse Computation in Verification26 Dec
  6. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts18 Nov
  7. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference11 Oct
  8. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding29 Aug
  9. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts17 Jun
  10. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE26 May

Other references · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  10. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  11. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  12. 2024AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structurescited by 18
5 more
  1. 2026LK Losses: Direct Acceptance Rate Optimization for Speculative Decodingcited by 18
  2. 2024Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generationcited by 16
  3. 2025TETRIS: Optimal Draft Token Selection for Batch Speculative Decodingcited by 15
  4. 2026SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decodingcited by 10
  5. 2025Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decodingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fillcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding
  2. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  3. 2026SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
  4. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices
  5. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  6. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
  7. 2026Margins, Not Windows: Training-Free Per-Step Lossy Speculative Decoding
  8. 2026A Practical Investigation of Training-free Relaxed Speculative Decoding