# The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts

arXiv: https://arxiv.org/abs/2609.22156 · HTML: https://mlsanity.com/paper/2609.22156/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 26 Aug 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)
- Efficient LLM inference and serving / [Diffusion LLM and MoE inference](https://mlsanity.com/inference/diffusion-llm-and-moe-inference/)

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024)
- [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE](https://mlsanity.com/paper/2505.19645/) (2025)
- [AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures](https://mlsanity.com/paper/2412.18910/) (2024)
- [LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding](https://mlsanity.com/paper/2602.23881/) (2026)
- [Draft Model Knows When to Stop: Self-Verification Speculative Decoding for Long-Form Generation](https://mlsanity.com/paper/2411.18462/) (2024)
- [TETRIS: Optimal Draft Token Selection for Batch Speculative Decoding](https://mlsanity.com/paper/2502.15197/) (2025)
- [Utility-Driven Speculative Decoding for Mixture-of-Experts](https://mlsanity.com/paper/2506.20675/) (2025)
- [SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference](https://mlsanity.com/paper/2510.10302/) (2025)
- [MoE-Spec: Expert Budgeting for Efficient Speculative Decoding](https://mlsanity.com/paper/2602.16052/) (2026)
- [MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts](https://mlsanity.com/paper/2511.14102/) (2025)
- [SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding](https://mlsanity.com/paper/2604.09557/) (2026)
- [Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding](https://mlsanity.com/paper/2508.21706/) (2025)
- [Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding](https://mlsanity.com/paper/2504.00030/) (2025)
- [SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding](https://mlsanity.com/paper/2604.10152/) (2026)
- [Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding](https://mlsanity.com/paper/2605.00342/) (2026)
- [Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts](https://mlsanity.com/paper/2607.12696/) (2026)
- [Accelerate Speculative Decoding with Sparse Computation in Verification](https://mlsanity.com/paper/2512.21911/) (2025)

## Built on by (most cited first, 1 of 1)

- [AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fill](https://mlsanity.com/paper/2610.05774/) (2026)

## Abstract (arXiv)

Speculative decoding in Mixture-of-Experts (MoE) models faces the problem of unstable verification cost caused by input-dependent expert loading. To study the physics of this process, we formulate speculation-budget selection as an offline Stochastic Shortest Path (SSP) problem over reference sequences and build a diagnostic Oracle that uses counterfactual simulation to account for MoE verification cost. A detailed analysis of the Oracle's decisions on the Qwen3-Coder and EAGLE-3 pairing, in the space of marginal deltas (Delta Space), shows that rejected candidates form a strict linear boundary. This result demonstrates that a complex global optimization is locally governed by a necessary condition balancing marginal cost against expected progress (fracDelta mathbbE[Cost]Delta mathbbE[a]), providing a rigorous mathematical reference point for designing future adaptive online heuristics.