Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Study
On the map: Mixture-of-experts → Efficient MoE training and inference systems
Abstract (arXiv)
Serving a 235B-parameter Mixture-of-Experts (MoE) model on a single 8 GB GPU is bottlenecked not by compute but by memory bandwidth: decode must stream each token's active experts from whichever tier holds them, and on consumer hardware most experts sit on an SSD far slower than RAM. We quantify this bandwidth wall on Qwen3-235B (Q4_K_M, 134 GB): measured decode is 0.44 tok/s warm, matching a bytes-per-token / bandwidth model, while a batching scheme that should amortize one disk sweep instead collapses at batch 32 from paging thrash. We build llama-moe-trace, a zero-surgery router-telemetry tool, and measure routing on Qwen3-30B: adjacent-token expert reuse is 2.0x chance, 95% of traffic uses 52.5% of experts, and an LRU cache of 13.4% of experts serves 66% of requests. We then ask whether cacheability is trainable: we pre-register training of 137M MoE language models with auxiliary locality and domain router losses, under joint criteria on cache-miss reduction and perplexity. The mechanism works (misses down up to 60%; a 99% static-pin hit rate) but every configuration fails the pre-registered <=1% perplexity gate -- miss reduction and quality are tightly coupled. Concurrent StickyMoE reports the same loss as near-free on single-domain sub-25M models; on multi-domain 137M we find the tax real. Our contribution is this pre-registered, stricter-criterion, multi-domain evaluation plus edge-serving measurements. A 340M rung shows the tax does not shrink with scale (it rises slightly). We further show training-free cache-aware rerouting stacks with trained locality -- together ~80% miss reduction at <=3.4% perplexity at both sizes, far cheaper than either alone -- while domain-primed prefetching does not help. All code, traces, and the pre-registration are released.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 30 of its 36 references that are in the map. First posted on arXiv 18 Aug 2026.
Builds on
30 of 36 references.
Closest ancestors · 13
Same direction, newest first
- 2026ELDR: Expert-Locality-Aware Decode Routing for PD-Disaggregated MoE Serving1 Jul
- 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference26 May
- 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models30 Jan
- 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gate17 Feb
- 2024A Survey on Inference Optimization Techniques for Mixture of Experts Models18 Dec
- 2024DAOP: Data-Aware Offloading and Predictive Pre-Calculation for Efficient MoE Inference16 Dec
- 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference27 Nov
- 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
- 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling23 Oct
- 2024AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference19 Aug
- 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
- 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
Other references · 17
- 2025Qwen3 Technical Reportcited by 8,195
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
- 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
- 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
- 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
- 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
- 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
5 more
- 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
- 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 101
- 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
- 2026Three Phases of Expert Routing: How Load Balance Evolves During Mixture-of-Experts Trainingcited by 4
- 2026Sticky Routing: Training MoE Models for Memory-Efficient Inferencecited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026WiSP: A Working-Set View of Mixture-of-Experts Serving on Extremely Low-Resource Hardware
- 2025In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading
- 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
- 2026Calibrate, Then Route: A Measured Study of Learned Request Routing for Disaggregated LLM Serving
- 2025Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens
- 2026Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging
- 2025SMoE: An Algorithm-System Co-Design for Pushing MoE to the Edge via Expert Substitution
- 2026The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction