mlsanityData to 14 Oct 2026

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

arXiv 2608.11688 ↗First posted 12 Aug 2026cs.AR cs.AI cs.LGcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models are attractive for edge deployment because they provide high model capacity while activating only a small subset of parameters per token, improving compute efficiency. However, MoE inference at the edge is fundamentally limited by memory. Expert parameters are large and often reside in off-chip memory due to capacity, cost, and power constraints, putting expert loading to the critical path. We present APEX: Adaptive Expert Prefetching, a predictive resource management framework that overlaps expert loading with useful computation. APEX introduces a lightweight prefetch router that predicts candidate experts before the attention block to dynamically fetch additional experts using a learned confidence model. This adaptive strategy achieves over 99% overlap accuracy, significantly outperforming fixed top-k prefetching techniques. APEX supports two execution modes: a correctness-preserving mode that guarantees exact routing semantics, and a stall-free mode that eliminates residual stalls by operating on available experts with negligible impact on application accuracy. Across multiple MoE models, the correctness-preserving mode reduces per-token latency by up to 26% and improves energy-delay product (EDP) by up to 41% over state-of-the-art baselines, while the stall-free mode provides additional efficiency gains with negligible impact on application accuracy. These results establish adaptive, confidence-driven expert prefetching as an effective approach for efficient MoE inference on edge systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 39 of its 56 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

39 of 56 references.

Closest ancestors · 10

Same direction, newest first

  1. 2026MoE-SpAc: Efficient MoE Inference Based on Speculative Activation Utility in Heterogeneous Edge Scenarios12 Feb
  2. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate17 Feb
  3. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  4. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching29 Oct
  5. 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
  6. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  7. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  8. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  9. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  10. 2021FastMoE: A Fast Mixture-of-Expert Training System24 Mar

Other references · 29

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2024Mixtral of Expertscited by 1,823
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
17 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  2. 2022Efficiently Scaling Transformer Inferencecited by 539
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  4. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  6. 2024A Review on Edge Large Language Models: Design, Execution, and Applicationscited by 100
  7. 2024AI and Memory Wallcited by 92
  8. 2023Ramulator 2.0: A Modern, Modular, and Extensible DRAM Simulatorcited by 30
  9. 2023Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximationcited by 8
  10. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specializationcited by 7
  11. 2025UbiMoE: A Ubiquitous Mixture-of-Experts Vision Transformer Accelerator With Hybrid Computation Pattern on FPGAcited by 4
  12. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Modelscited by 4
  13. 2024MFIT: Multi-Fidelity Thermal Modeling for 2.5D and 3D Multi-Chiplet Architecturescited by 2
  14. 2026DUET: Disaggregated Hybrid Mamba-Transformer LLMs with Prefill and Decode-Specific Packagescited by 2
  15. 2025THERMOS: Thermally-Aware Multi-Objective Scheduling of AI Workloads on Heterogeneous Multi-Chiplet PIM Architecturescited by 1
  16. 2025CHIPSIM: A Co-Simulation Framework for Deep Learning on Chiplet-Based Systemscited by 1
  17. 2026LEXI: Lossless Exponent Coding for Efficient Inter-Chiplet Communication in Hybrid LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
  2. 2025ExpertFlow: Adaptive Expert Scheduling and Memory Coordination for Efficient MoE Inference
  3. 2026SPICE: Speculative Prefetching with Low-Rank Expert Surrogates and Heterogeneous Orchestration for MoE Inference Acceleration
  4. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
  5. 2025LayerScope: Predictive Cross-Layer Scheduling for Efficient Multi-Batch MoE Inference on Legacy Servers
  6. 2026A Scheduling Framework for Efficient MoE Inference on Edge GPU-NDP Systems
  7. 2026Speculating Experts Accelerates Inference for Mixture-of-Experts
  8. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging