# MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE arXiv: https://arxiv.org/abs/2505.19645 ยท HTML: https://mlsanity.com/paper/2505.19645/ ## Key facts - Cited by 19 papers in the map (data through 14 Oct 2026). - Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. - Builds on 32 of its 47 references that are in the map. - First posted on arXiv 26 May 2025. ## Placed in - Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/) - Efficient LLM inference and serving / [Speculative decoding](https://mlsanity.com/inference/speculative-decoding/) ## Builds on (most cited first, 30 of 32) - [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023) - [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024) - [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022) - [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017) - [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021) - [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022) - [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020) - [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024) - [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023) - [DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models](https://mlsanity.com/paper/2401.06066/) (2024) - [Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads](https://mlsanity.com/paper/2401.10774/) (2024) - [EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty](https://mlsanity.com/paper/2401.15077/) (2024) - [EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees](https://mlsanity.com/paper/2406.16858/) (2024) - [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023) - [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025) - [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024) - [REST: Retrieval-Based Speculative Decoding](https://mlsanity.com/paper/2311.08252/) (2023) - [Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy](https://mlsanity.com/paper/2310.01334/) (2023) - [MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding](https://mlsanity.com/paper/2408.11049/) (2024) - [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://mlsanity.com/paper/2408.12757/) (2024) - [MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router](https://mlsanity.com/paper/2410.12013/) (2024) - [MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decomposition](https://mlsanity.com/paper/2411.01016/) (2024) - [HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference](https://mlsanity.com/paper/2411.01433/) (2024) - [SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices](https://mlsanity.com/paper/2406.02532/) (2024) - [LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation](https://mlsanity.com/paper/2408.15881/) (2024) - [QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models](https://mlsanity.com/paper/2310.16795/) (2023) - [AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference](https://mlsanity.com/paper/2408.10284/) (2024) - [STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning](https://mlsanity.com/paper/2409.06211/) (2024) ## Built on by (most cited first, 19 of 19) - [SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding](https://mlsanity.com/paper/2604.09557/) (2026) - [MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios](https://mlsanity.com/paper/2603.09983/) (2026) - [SpecVLM: Fast Speculative Decoding in Vision-Language Models](https://mlsanity.com/paper/2509.11815/) (2025) - [Fast LLM Post-training via Decoupled and Fastest-of-N Speculation](https://mlsanity.com/paper/2511.16193/) (2025) - [Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding](https://mlsanity.com/paper/2605.00342/) (2026) - [Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts](https://mlsanity.com/paper/2607.12696/) (2026) - [Online Draft Co-Training for Speculative Decoding in Large-Scale, Long-Context RL Post-Training](https://mlsanity.com/paper/2609.07108/) (2026) - [Accelerating Speculative Diffusions via Block Verification](https://mlsanity.com/paper/2606.13426/) (2026) - [DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference](https://mlsanity.com/paper/2607.24434/) (2026) - [S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices](https://mlsanity.com/paper/2608.15018/) (2026) - [AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding](https://mlsanity.com/paper/2608.02989/) (2026) - [The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Experts](https://mlsanity.com/paper/2609.22156/) (2026) - [ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Serving](https://mlsanity.com/paper/2604.14626/) (2026) - [Think Before You Grid-Search: Floor-First Triage for LLM Serving](https://mlsanity.com/paper/2607.05876/) (2026) - [EdgeXpert: An Edge Device for Memory-Efficient LLM Inference with Mixture-of-Experts and Speculative Decoding](https://mlsanity.com/paper/2608.05303/) (2026) - [Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation](https://mlsanity.com/paper/2609.22471/) (2026) - [CAST: Cost-Aware Speculative Trees from One-Pass Block Drafters](https://mlsanity.com/paper/2610.00321/) (2026) - [AdaSpark: Adaptive DSpark with Online Learning for Tree Verification and N-gram Fill](https://mlsanity.com/paper/2610.05774/) (2026) - [Cascadia: Resident 975B MoE Inference on Eleven AI PCs](https://mlsanity.com/paper/2610.07219/) (2026) ## Abstract (arXiv) Large Language Models (LLMs) have achieved remarkable success across many applications, with Mixture of Experts (MoE) models demonstrating great potential. Compared to traditional dense models, MoEs achieve better performance with less computation. Speculative decoding (SD) is a widely used technique to accelerate LLM inference without accuracy loss, but it has been considered efficient only for dense models. In this work, we first demonstrate that, under medium batch sizes, MoE surprisingly benefits more from SD than dense models. Furthermore, as MoE becomes sparser -- the prevailing trend in MoE designs -- the batch size range where SD acceleration is expected to be effective becomes broader. To quantitatively understand tradeoffs involved in SD, we develop a reliable modeling based on theoretical analyses. While current SD research primarily focuses on improving acceptance rates of algorithms, changes in workload and model architecture can still lead to degraded SD acceleration even with high acceptance rates. To address this limitation, we introduce a new metric 'target efficiency' that characterizes these effects, thus helping researchers identify system bottlenecks and understand SD acceleration more comprehensively. For scenarios like private serving, this work unveils a new perspective to speed up MoE inference, where existing solutions struggle. Experiments on different GPUs show up to 2.29x speedup for Qwen2-57B-A14B at medium batch sizes and validate our theoretical predictions.