mlsanityData to 7 Oct 2026

Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation

arXiv 2609.22471 ↗First posted 18 Sep 2026cs.LG cs.CLcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Mixture-of-Experts (MoE) models are increasingly deployed alongside Speculative Decoding (SD) to accelerate inference, but combining the two is challenging. SD improves the inference speed of dense models by verifying groups of tokens in parallel. However, the inference speedup for SD with MoEs depends heavily on the number of tokens being verified. Using more verification tokens results in more experts being transferred from DRAM to the Neural Processing Unit (NPU), which increases the memory transfer cost. This negatively impacts model runtime, as memory transfer is typically the bottleneck in inference. In this work, we investigate the impact of MoE router design during training on the speed of MoEs with SD. We find that routers with high degrees of expert coactivation result in much faster runtimes, mitigating the impact of using more verification tokens. Motivated by this observation, we assess the impact of various router design choices on expert coactivation and runtime using billion-parameter transformer models. We find that combining a global load-balancing loss, shared experts, a consistency loss, and an autoregressive expert selection mechanism during training results in significantly stronger expert coactivation. This increased coactivation translates into higher overall runtime throughput: our exploration yields a model that improves throughput by 21% over MoE baselines, while maintaining on-par accuracy with the baseline MoE.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 26 of its 30 references that are in the map. First posted on arXiv 18 Sep 2026.

Builds on

26 of 30 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study18 Aug
  2. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices15 Aug
  3. 2026AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding4 Aug
  4. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference27 Jul
  5. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts14 Jul
  6. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding17 Feb
  7. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts17 Jun
  8. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE26 May
  9. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models21 Jan
  10. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference27 Nov

Other references · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  8. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  12. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
4 more
  1. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  2. 2025Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficientcited by 29
  3. 2025Instruction-Following Pruning for Large Language Modelscited by 14
  4. 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inferencecited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  2. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  3. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts
  4. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding
  5. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios
  6. 2026BASE: Batch-Aware Selection of Experts Using Predicted Removal Error for Efficient MoE Decoding
  7. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  8. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inference