mlsanityData to 14 Oct 2026

A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inference

arXiv 2606.15453 ↗First posted 13 Jun 2026cs.AR cs.LGcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) based large language models (LLMs), such as Qwen and DeepSeek, have recently emerged as an effective approach to improving model capacity without proportionally increasing computational cost. By replacing the conventional feed-forward network in dense LLMs with a set of experts and activating only a subset of them for each input token, MoE models significantly increase the total number of parameters while keeping the per-token computation relatively manageable. However, this dynamic and irregular expert activation pattern also introduces substantial expert loading overhead during inference, since the required experts must be fetched on demand according to token-dependent routing results. As a result, expert loading latency becomes a major source of performance and energy inefficiency. To this end, we first perform a comprehensive analysis of expert selection behavior in various MoE-based LLMs and applications, including language understanding and code generation. Our analysis reveals that, within each application domain, expert requests exhibit strong correlation across both adjacent MoE layers and consecutive decoding tokens, making future expert activations predictable. Based on this insight, we propose ST-MoE, a spatio-temporal expert prefetching framework that proactively stages experts ahead of use to overlap expert loading with ongoing computation. ST-MoE combines a lightweight runtime prediction mechanism that preserves the original routing behavior with a reconfigurable hardware design that efficiently supports dynamic expert prefetching. The combined effect of the prediction mechanism with the supporting hardware significantly improves MoE inference performance and energy efficiency while preserving model inference accuracy.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 20 of its 42 references that are in the map. First posted on arXiv 13 Jun 2026.

Builds on

20 of 42 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate17 Feb
  2. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  3. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug
  4. 2022Tutel: Adaptive Mixture-of-Experts at Scale7 Jun

Other references · 16

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2015Teaching Machines to Read and Comprehendcited by 1,049
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  6. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  7. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  8. 2023Using an LLM to Help With Code Understandingcited by 228
  9. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  10. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  11. 2023BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questionscited by 125
  12. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
4 more
  1. 2016A Thorough Examination of the CNN/Daily Mail Reading Comprehension Taskcited by 72
  2. 2023Bias Testing and Mitigation in LLM-based Code Generationcited by 53
  3. 2024Understanding Large-Language Model (LLM)-powered Human-Robot Interactioncited by 43
  4. 2024BitMoD: Bit-serial Mixture-of-Datatype LLM Accelerationcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inferencecited by 3
  2. 2026Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Expertscited by 2
  3. 2026SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloadingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading
  2. 2025Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models
  3. 2025Faster MoE LLM Inference for Extremely Large Models
  4. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  5. 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
  6. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
  7. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  8. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference