mlsanityData to 14 Oct 2026

Reproducible Evaluation of MoE Expert Caching: Replay Semantics, Workload Contamination, and Operating Regimes

arXiv 2608.07911 ↗First posted 8 Aug 2026cs.LG cs.PFcited by 0

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Mixture-of-Experts (MoE) models have outgrown accelerator memory, and offloading expert weights to host memory is now standard. This makes expert cache management an attractive lever: a policy that raised the hit rate would cut expert traffic per token. Evaluating that is a measurement problem, and we find the measurement fragile. With a trace-driven, event-atomic simulator over three MoE models (40, 64, 128 experts), we isolate three evaluation axes that change conclusions, not just numbers. Replay semantics: under a fused-event traffic contract, an inconsistent per-access replay inflates recency-based policies by 27-29% while leaving frequency-based and static ones within 4%, inverting the policy ranking. Workload contamination: probe sets using one instruction template per category produce verbatim-identical generation prefixes; a matched-pair rendering intervention moves the measured early-window effect by 19.4-31.9 points and reverses which workloads look most cache-friendly. Operating regimes: normalized miss fractions do not transfer across models, so the per-step expert union relative to per-layer capacity must be reported -- yet permuting only the temporal order of an identical event stream moves the offline-optimal gap from 44.9% to 30.8%, so it is not sufficient. Corrected, a stable gap to the offline optimum remains (44.2-45.9% over 13 frozen workload compositions). A forced-admission oracle attributes 84.3-96.6% of it to knowing which resident expert is used furthest in the future. A causal next-use predictor, used as an eviction rule, recovers -11.4% of the gap; it picks an optimal victim 3.4% of the time, against 2.4% for a random resident block and 20.6-22.1% for LRU and LFRU. Our position is narrow: in our evaluated settings a large offline-optimal gap substantially overstates the gains recovered by representative lightweight causal mechanisms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 14 of its 24 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

14 of 24 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026SpecMD: A Comprehensive Study On Speculative Expert Prefetching3 Feb
  2. 2025SP-MoE: Speculative Decoding and Prefetching for Accelerating MoE-based Model Inference11 Oct
  3. 2025MoE-Beyond: Learning-Based Expert Activation Prediction on Edge Devices23 Aug
  4. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate17 Feb
  5. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  6. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  7. 2023Fast Inference of Mixture-of-Experts Language Models with Offloading28 Dec
  8. 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference23 Aug

Other references · 6

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  6. 2024Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflowscited by 232

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading
  2. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study
  3. 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
  4. 2026Flattening Every Memory Peak in Long-Context Mixture-of-Experts Training
  5. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
  6. 2026Dynamic HBM Repartitioning for Multi-Turn MoE Serving
  7. 2026DALI: A Workload-Aware Offloading Framework for Efficient MoE Inference on Local PCs
  8. 2026SeqMoE: Toward Full-Load Performance via Predictive and Graph-Compatible MoE Offloading