mlsanityData to 14 Oct 2026

SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding

arXiv 2604.10152 ↗First posted 11 Apr 2026cs.AI cs.LGcited by 4

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

The Mixture-of-Experts (MoE) architecture has emerged as a promising approach to mitigate the rising computational costs of large language models (LLMs) by selectively activating parameters. However, its high memory requirements and sub-optimal parameter efficiency pose significant challenges for efficient deployment. Although CPU-offloaded MoE inference systems have been proposed in the literature, they offer limited efficiency, particularly for large batch sizes. In this work, we propose SpecMoE, a memory-efficient MoE inference system based on our self-assisted speculative decoding algorithm. SpecMoE demonstrates the effectiveness of applying speculative decoding to MoE inference without requiring additional model training or fine-tuning. Our system improves inference throughput by up to 4.30×, while significantly reducing bandwidth requirements of both memory and interconnect on memory-constrained systems.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 43 of its 55 references that are in the map. First posted on arXiv 11 Apr 2026.

Builds on

43 of 55 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Duplex: A Device for Large Language Models with Mixture of Experts, Grouped Query Attention, and Continuous Batching2 Sep
  2. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  3. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun
  5. 2022MoESys: A Distributed and Efficient Mixture-of-Experts Training and Inference System for Internet Services20 May
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  11. 2024Mixtral of Expertscited by 1,823
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
25 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  5. 2015Teaching Machines to Read and Comprehendcited by 1,049
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  12. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  13. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  14. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  15. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  16. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  17. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  18. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  19. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  20. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  21. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  22. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  23. 2023SPEED: Speculative Pipelined Execution for Efficient Decodingcited by 47
  24. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  25. 2023Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inferencecited by 37

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devicescited by 2
  2. 2026The Limits of Speculation: Bounding Speculative Decoding in Mixture-of-Expertscited by 1
  3. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0
  4. 2026Cascadia: Resident 975B MoE Inference on Eleven AI PCscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
  2. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding
  3. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
  4. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts
  5. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
  6. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  7. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  8. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts