mlsanityData to 14 Oct 2026

In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading

arXiv 2511.05814 ↗First posted 8 Nov 2025cs.LG cs.AIcited by 2

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

In today's landscape, Mixture of Experts (MoE) is a crucial architecture that has been used by many of the most advanced models. One of the major challenges of MoE models is that they usually require much more memory than their dense counterparts due to their unique architecture, and hence are harder to deploy in environments with limited GPU memory, such as edge devices. MoE offloading is a promising technique proposed to overcome this challenge, especially if it is enhanced with caching and pre-fetching, but prior work stopped at suboptimal caching algorithm and offered limited insights. In this work, we study MoE offloading in depth and make the following contributions: 1. We analyze the expert activation and LRU caching behavior in detail and provide traces. 2. We propose LFU caching optimization based on our analysis and obtain strong improvements from LRU. 3. We implement and experiment speculative expert pre-fetching, providing detailed trace showing its huge potential . 4. In addition, our study extensively covers the behavior of the MoE architecture itself, offering information on the characteristic of the gating network and experts. This can inspire future work on the interpretation of MoE models and the development of pruning techniques for MoE architecture with minimal performance loss.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 10 of its 12 references that are in the map. First posted on arXiv 8 Nov 2025.

Builds on

10 of 12 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  2. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 8

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2024Mixtral of Expertscited by 1,823
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  8. 2023Boosted Prompt Ensembles for Large Language Modelscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
  2. 2026OLED-MoE: Accelerating MoE-Based dLLM Inference via Inter-Iteration Locality-Aware Expert Offloadingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026SpecMD: A Comprehensive Study On Speculative Expert Prefetching
  2. 2026SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading
  3. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  4. 2025OD-MoE: On-Demand Expert Loading for Cacheless Edge-Distributed MoE Inference
  5. 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
  6. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Caching
  7. 2026Reproducible Evaluation of MoE Expert Caching: Replay Semantics, Workload Contamination, and Operating Regimes
  8. 2025MoBiLE: Efficient Mixture-of-Experts Inference on Consumer GPU with Mixture of Big Little Experts