mlsanityData to 14 Oct 2026

SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference

arXiv 2510.10302 ↗First posted 11 Oct 2025cs.DCcited by 13

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

The Mixture-of-Experts (MoE) architecture has been widely adopted in large language models (LLMs) to reduce computation cost through model sparsity. Employing speculative decoding (SD) can further accelerate MoE inference by drafting multiple tokens per step and verifying them in parallel. However, combining MoE with SD inflates GPU memory and aggravates CPU-GPU bandwidth contention during multi-token verification. Existing MoE offloading systems are SD-agnostic and do not address this bottleneck. We present SP-MoE, the first SD-aware expert-offloading and compute-communication pipelining framework. SP-MoE introduces: (1) speculative expert prefetching that exploits structural correspondence between the draft and target models to prefetch likely experts ahead of verification; (2) a cutoff-layer policy that bounds per-layer prefetch depth based on empirical profiles and an analytical latency model, guaranteeing just-in-time availability without overfetch; and (3) a pipelined runtime with asynchronous prefetch threads and batched I/O to hide loading latency. Extensive experiments demonstrate that SP-MoE achieves a 1.07-3.5 times TPOT speedup over state-of-the-art methods across diverse datasets, environments, and MoE-based models.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 26 of its 44 references that are in the map. First posted on arXiv 11 Oct 2025.

Builds on

26 of 44 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  2. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  3. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  4. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  5. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  6. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 20

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  7. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  9. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  10. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  11. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
  12. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
8 more
  1. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  2. 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 252
  3. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  4. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 102
  5. 2024Kangaroo: Lossless Self-Speculative Decoding via Double Early Exitingcited by 58
  6. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  7. 2024Generation Meets Verification: Accelerating Large Language Model Inference with Smart Parallel Auto-Correct Decodingcited by 22
  8. 2024HumanEval on Latest GPT Models -- 2024cited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decodingcited by 12
  2. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 4
  3. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Expertscited by 3
  4. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inferencecited by 2
  5. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devicescited by 2
  6. 2026DFlow: Enabling Verifier Information Flow in Block Diffusion Speculative Decodingcited by 2
  7. 2026Speculating Experts Accelerates Inference for Mixture-of-Expertscited by 1
  8. 2026AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decodingcited by 1
  9. 2026The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Expertscited by 1
  10. 2026ELMoE-3D: Leveraging Intrinsic Elasticity of MoE for Hybrid-Bonding-Enabled Self-Speculative Decoding in On-Premises Servingcited by 0
  11. 2026Reproducible Evaluation of MoE Expert Caching: Replay Semantics, Workload Contamination, and Operating Regimescited by 0
  12. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0
1 more
  1. 2026SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts
  2. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  3. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts
  4. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  5. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding
  6. 2026SPICE: Speculative Prefetching with Low-Rank Expert Surrogates and Heterogeneous Orchestration for MoE Inference Acceleration
  7. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
  8. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE