mlsanityData to 14 Oct 2026

Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference

arXiv 2609.04895 ↗First posted 4 Sep 2026cs.CLcited by 3

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture-of-Experts (MoE) models activate few experts per token, yet their full expert sets can exceed GPU memory and require repeated weight transfers during decoding. We formulate expert-cache management as a model-side algorithmic problem and propose cache-aware post-training that jointly adapts the MoE backbone and lightweight auxiliary routers while preserving the native inference-time Top-K rule. The update-only Temporal Router learns same-layer retention across tokens without proactive loading. The full Spatio-Temporal Router adds a Spatio Router that uses the causal predecessor's hidden state to refine the temporal cache before target-layer access. We evaluate both modes on Qwen3 and GPT-OSS across GSM8K, MATH, and CommonsenseQA. Temporal Router consistently improves hit rate and reduces expert-weight traffic over matched LM-only baselines. On Qwen3, the full mode improves adjusted hit rate by 1.15--18.03 points and reduces traffic by 4.6--53.3% relative to the strongest evaluated prefetching baseline; GPT-OSS results are competitive but task-dependent. Auxiliary-only training preserves baseline accuracy but yields modest coverage gains; joint post-training achieves substantially higher coverage. Sensitivity analyses distinguish the effects of cache capacity, refinement budget, and cache-loss weight on coverage, traffic, and quality.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 14 of its 21 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

14 of 21 references.

References · 14

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  7. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  8. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
  9. 2024ProMoE: Fast MoE-based LLM Serving using Proactive Cachingcited by 40
  10. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloadingcited by 24
  11. 2026SpecMD: A Comprehensive Study On Speculative Expert Prefetchingcited by 3
  12. 2026Temporally Extended Mixture-of-Experts Modelscited by 3
2 more
  1. 2026A Spatio-Temporal Expert Prefetching Framework for Efficient MoE-based LLM Inferencecited by 3
  2. 2026MoEless: Efficient MoE LLM Serving via Serverless Computingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Efficient Mixture-of-Experts with Speculative Decoding via Expert Coactivationcited by 0
  2. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0
  3. 2026PA1-OPD: One-Step Prefix-Augmented On-Policy Distillation for Language Model Pretrainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Sticky Routing: Training MoE Models for Memory-Efficient Inference
  2. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
  3. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference
  4. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
  5. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models
  6. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
  7. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
  8. 2026APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference