mlsanityData to 7 Oct 2026

BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devices

arXiv 2609.14643 ↗First posted 13 Sep 2026cs.AR cs.DCcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

also in 1 more
Abstract (arXiv)

Mixture-of-Experts (MoE) models expand language model capacity on smartphones, but expert offloading remains constrained by limited DRAM capacity and costly data movement. Sequential token routing couples expert execution to fragmented flash reads and multistage NPU preparation, leaving sparse computation stalled on weight transfers. Each transfer serves few tokens before execution moves on. We exploit the multi-token verification window of speculative decoding to decouple expert movement from single-token execution, enabling weight reuse, contiguous flash reads, and load-compute overlap. We present textscBigMoMo, a mobile MoE runtime that exploits this window across the memory hierarchy. It prunes speculative branches and expert activations using acceptance rates, routing impact, and movement cost; reorganizes on-flash experts according to runtime co-loading patterns; and batches ready experts to overlap NPU computation with pending transfers. Across four MoE models and five benchmarks on two mobile platforms, textscBigMoMo achieves mean decoding speedups of 4.83× over on-demand autoregressive offloading and 1.82× over the best speculative MoE baseline, supporting MoE models up to 30B parameter.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 36 of its 45 references that are in the map. First posted on arXiv 13 Sep 2026.

Builds on

36 of 45 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026S2-MoE: Enabling Efficient Self-Speculative Decoding for Mixture-of-Experts on Edge Devices15 Aug
  2. 2026MobileMoE: Scaling On-Device Mixture of Experts26 May
  3. 2026SpecMoE: A Fast and Efficient Mixture-of-Experts Inference via Self-Assisted Speculative Decoding11 Apr
  4. 2026MoE-Spec: Expert Budgeting for Efficient Speculative Decoding17 Feb
  5. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios12 Feb
  6. 2025MoE-SpeQ: Speculative Quantized Decoding with Proactive Expert Prefetching and Offloading for Mixture-of-Experts18 Nov
  7. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference11 Oct
  8. 2025Accelerating Mixture-of-Experts Inference by Hiding Offloading Latency with Speculative Decoding29 Aug
  9. 2025Utility-Driven Speculative Decoding for Mixture-of-Experts17 Jun
  10. 2025HybriMoE: Hybrid CPU-GPU Scheduling and Cache Management for Efficient MoE Inference8 Apr
  11. 2025Klotski: Efficient Mixture-of-Expert Inference via Expert-Aware Multi-Batch Pipeline9 Feb
  12. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
4 more
  1. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference27 Nov
  2. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  3. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  4. 2023EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices28 Aug

Other references · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  6. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  8. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  9. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  10. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  11. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  12. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
8 more
  1. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  2. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  3. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 74
  4. 2024Fast On-device LLM Inference with NPUscited by 58
  5. 2024MobiLlama: Towards Accurate and Lightweight Fully Transparent GPTcited by 34
  6. 2024Apple Intelligence Foundation Language Modelscited by 22
  7. 2025Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inferencecited by 20
  8. 2024Fox-1: Open Small Language Model for Cloud and Edgecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DraftExpert: Expansion-Aware Self-Speculative Decoding for End-Device MoE Inference
  2. 2026Lever: Speculative LLM Inference on Smartphones
  3. 2026Paging the Experts: A Reproducible Characterization of Flash-Backed MoE Inference on iPhone
  4. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivation
  5. 2026AcceptMoE: Commitment-Weighted Self-Sizing Verifier Expert Sets for Efficient MoE Speculative Decoding
  6. 2025MoESD: Unveil Speculative Decoding's Potential for Accelerating Sparse MoE
  7. 2026FlashMoE: Reducing SSD I/O Bottlenecks via ML-Based Cache Replacement for Mixture-of-Experts Inference on Edge Devices
  8. 2026Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts