# S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices

arXiv: https://arxiv.org/abs/2608.15018 · HTML: https://mlsanity.com/paper/2608.15018/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts.
- Builds on 24 of its 39 references that are in the map.
- First posted on arXiv 15 Aug 2026.

## Placed in

- Mixture-of-experts / [Efficient MoE training and inference systems](https://mlsanity.com/moe/efficient-moe-training-and-inference-systems/)
- Efficient LLM inference and serving / [Diffusion LLM and MoE inference](https://mlsanity.com/inference/diffusion-llm-and-moe-inference/)

## Builds on (most cited first, 24 of 24)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Fast Inference from Transformers via Speculative Decoding](https://mlsanity.com/paper/2211.17192/) (2022)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model](https://mlsanity.com/paper/2405.04434/) (2024)
- [Mixture-of-Experts with Expert Choice Routing](https://mlsanity.com/paper/2202.09368/) (2022)
- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test](https://mlsanity.com/paper/2503.01840/) (2025)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding](https://mlsanity.com/paper/2401.07851/) (2024)
- [MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE](https://mlsanity.com/paper/2505.19645/) (2025)
- [QSpec: Speculative Decoding with Complementary Quantization Schemes](https://mlsanity.com/paper/2410.11305/) (2024)
- [Utility-Driven Speculative Decoding for Mixture-of-Experts](https://mlsanity.com/paper/2506.20675/) (2025)
- [SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference](https://mlsanity.com/paper/2510.10302/) (2025)
- [ProPD: Dynamic Token Tree Pruning and Generation for LLM Parallel Decoding](https://mlsanity.com/paper/2402.13485/) (2024)
- [MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts](https://mlsanity.com/paper/2511.14102/) (2025)
- [Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding](https://mlsanity.com/paper/2508.21706/) (2025)
- [MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios](https://mlsanity.com/paper/2603.09983/) (2026)
- [SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding](https://mlsanity.com/paper/2604.10152/) (2026)
- [Rethinking the Role of Tensor Decompositions in Post-Training LLM Compression](https://mlsanity.com/paper/2606.03465/) (2026)

## Built on by (most cited first, 2 of 2)

- [BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devices](https://mlsanity.com/paper/2609.14643/) (2026)
- [Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation](https://mlsanity.com/paper/2609.22471/) (2026)

## Abstract (arXiv)

Deploying large language models (LLMs) for inference on edge devices is challenging due to severe memory and bandwidth constraints. While speculative decoding and Mixture-of-Experts (MoE) have been proposed to improve inference efficiency, naively combining them often incurs excessive verification overhead and poor expert reuse, limiting their effectiveness in memory-bound edge settings. In this work, we propose S2-MoE, an efficient self-speculative decoding framework for MoE inference on edge devices. S2-MoE reduces redundant verification through routing-aware adaptive speculative expansion, improves verification efficiency with reuse-aware expert gating, and aligns draft and target execution via shared context. Implemented in llama.cpp, S2-MoE achieves up to 5.3× speedup (about 2.0× on average) over standard autoregressive decoding across diverse MoE models and datasets on edge devices. Code is available at https://github.com/angerybob/S2-MoE.