mlsanityData to 7 Oct 2026

MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios

arXiv 2603.09983 ↗First posted 12 Feb 2026cs.LG cs.AI cs.CLcited by 5

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models enable scalable performance but face severe memory constraints on edge devices. Existing offloading strategies struggle with I/O bottlenecks due to the dynamic, low-information nature of autoregressive expert activation. In this paper, we propose to repurpose Speculative Decoding (SD) not merely as a compute accelerator, but as an informative lookahead sensor for memory management, supported by our theoretical and empirical analyses. Hence, we introduce MoE-SpAc, an MoE inference framework that integrates a Speculative Utility Estimator to track expert demand, a Heterogeneous Workload Balancer to dynamically partition computation via online integer optimization, and an Asynchronous Execution Engine to unify the prefetching and eviction in the same utility space. Extensive experiments on seven benchmarks demonstrate that MoE-SpAc achieves a 42% improvement in TPS over the SOTA SD-based baseline, and an average 4.04x speedup over all standard baselines. Code is available at https://github.com/lshAlgorithm/MoE-SpAc .

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 55 of its 76 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

55 of 76 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts18 Nov
  2. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers28 Sep
  3. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding29 Aug
  4. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE26 May
  5. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  6. 2025Mixture of Lookup Experts20 Mar
  7. 2025MoE-Gen: High-Throughput MoE Inference on a Single GPU with Module-Based Batching12 Mar
  8. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate17 Feb
  9. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  10. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  11. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  12. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
6 more
  1. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Service19 Jul
  2. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  3. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  4. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Models29 Oct
  5. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  6. 2022Accelerating Distributed MoE Training and Inference with Lina31 Oct

Other references · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
25 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  3. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  5. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  6. 2021Hash Layers For Large Sparse Modelscited by 236
  7. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  8. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  9. 2020The Lipschitz Constant of Self-Attentioncited by 171
  10. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  11. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  12. 2026Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Modelscited by 78
  13. 2018NVIDIA Tensor Core Programmability, Performance & Precisioncited by 77
  14. 2024MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Routercited by 58
  15. 2024MoE-I2: Compressing Mixture of Experts Models through Inter-Expert Pruning and Intra-Expert Low-Rank Decompositioncited by 57
  16. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 45
  17. 2024LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillationcited by 43
  18. 2023QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Modelscited by 42
  19. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  20. 2020Simple and Fast Algorithm for Binary Integer and Online Linear Programmingcited by 27
  21. 2024CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inferencecited by 21
  22. 2025Online Scheduling for LLM Inference with KV Cache Constraintscited by 17
  23. 2024Efficiency Unleashed: Inference Acceleration for LLM-based Recommender Systems with Speculative Decodingcited by 10
  24. 2026A Universal Load Balancing Principle and Its Application to Large Language Model Servingcited by 6
  25. 2025A Markov Categorical Framework for Language Modelingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devicescited by 2
  2. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inferencecited by 1
  3. 2026VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloadingcited by 0
  4. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inferencecited by 0
  5. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference
  2. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  3. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling
  4. 2026SPICE: Speculative Prefetching with Low-Rank Expert Surrogates and Heterogeneous Orchestration for MoE Inference Acceleration
  5. 2026The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
  6. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
  7. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding
  8. 2026SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding