mlsanityData to 7 Oct 2026

Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding

arXiv 2508.21706 ↗First posted 29 Aug 2025cs.DCcited by 8

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Recent advancements in Mixture of Experts (MoE) models have significantly increased their parameter scale as well as model performance. Extensive offloading techniques have been proposed to address the GPU memory limitations of MoE inference. However, due to the I/O bottleneck and sparse computation of MoE models, existing offloading techniques still suffer from low hardware utilization. To fully utilize the hardware resources, we propose SpecMoEOff, which employs the speculative decoding technique to enlarge the workload of each expert. SpecMoEOff orchestrates the GPU and CPU by both theoretical and empirical roofline analysis. In addition, we develop a dedicated CPU chunked attention verification kernel to fit the speculative decoding in offloading scenarios as well as minimizing the additional overhead led by draft models. SpecMoEOff further integrates an optimizer to automatically tune the hyperparameters of speculative decoding under given hardware and workload. Experimental results show that SpecMoEOff achieves up to 2.5x decode throughput improvement over the state-of-the-art MoE offloading techniques.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 21 of its 27 references that are in the map. First posted on arXiv 29 Aug 2025.

Builds on

21 of 27 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  2. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2024Mixtral of Expertscited by 1,785
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  11. 2015Teaching Machines to Read and Comprehendcited by 1,039
  12. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
7 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  5. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 318
  6. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 187
  7. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decodingcited by 9
  2. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarioscited by 5
  3. 2026Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decodingcited by 3
  4. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 3
  5. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devicescited by 2
  6. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inferencecited by 1
  7. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0
  8. 2026The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding
  2. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  3. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  4. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts
  5. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  6. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts
  7. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
  8. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts