mlsanityData to 14 Oct 2026

ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference

arXiv 2605.27081 ↗First posted 26 May 2026cs.LG cs.AI cs.DCcited by 3

On the map: Mixture-of-experts → Efficient MoE training and inference systems

Abstract (arXiv)

Fine-grained Mixture-of-Experts (MoE) models sparsely activate only a subset of experts per token, reducing activated computation while maintaining high model capacity. However, in memory-constrained inference scenarios, only a small set of experts can be cached. Experts not in the cache must be fetched from slow external storage (e.g., UFS), leading to frequent evictions and substantial I/O overhead. We propose ReMoE, a router fine-tuning framework designed to boost token-wise expert reuse. ReMoE biases the router toward recently selected experts, producing temporally stable routing that better matches cache locality constraints. By increasing short-horizon expert reuse, ReMoE reduces expert fetches from storage without adding inference-time computation. Experiments on DeepSeek and Qwen models show that ReMoE improves expert reuse by 26% while maintaining downstream task performance. Real-system evaluations further confirm these benefits, improving output throughput by 8.4% under vLLM GPU-CPU expert offloading and reducing TPOT by 43.6-49.8% under llama.cpp on Jetson Orin NX, corresponding to a 1.77-1.99× decode speedup across diverse workloads. Checkpoints and usage instructions are available at https://github.com/BUAA-OSCAR/ReMoE.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Efficient MoE training and inference systems, one of 12 directions in Mixture-of-experts. Builds on 37 of its 69 references that are in the map. First posted on arXiv 26 May 2026.

Builds on

37 of 69 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025In-depth Analysis on Caching and Pre-fetching in Mixture of Experts Offloading8 Nov
  2. 2025Taming Latency-Memory Trade-Off in MoE-Based LLM Serving via Fine-Grained Expert Offloading7 Feb
  3. 2024Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference27 Nov
  4. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs18 Nov
  5. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inference3 Nov
  6. 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models10 Feb
  7. 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cache25 Jan
  8. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Experts29 Nov
  9. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale14 Jan

Other references · 28

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
16 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  2. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  4. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  5. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  6. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  7. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 128
  8. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  9. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  10. 2025Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Modelscited by 46
  11. 2022Structural Pruning via Latency-Saliency Knapsackcited by 41
  12. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inferencecited by 37
  13. 2025TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applicationscited by 16
  14. 2024KVPR: Efficient LLM Inference with I/O-Aware KV Cache Partial Recomputationcited by 13
  15. 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Modelscited by 8
  16. 2025CoServe: Efficient Collaboration-of-Experts (CoE) Model Inference with Limited Memorycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Sticky Routing: Training MoE Models for Memory-Efficient Inferencecited by 2
  2. 2026Cacheable by Design? Training Mixture-of-Experts Routers for Locality Against the Edge Memory-Bandwidth Wall: A Pre-Registered Negative Result with a Systems Measurement Studycited by 1
  3. 2026MaskCoFT: Masked Co-Adaptive Fine-Tuning for Memory-Efficient MoE Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  2. 2025Accelerating Mixture-of-Expert Inference with Adaptive Expert Split Mechanism
  3. 2026MELINOE: Fine-Tuning Enables Memory-Efficient Inference for Mixture-of-Experts Models
  4. 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
  5. 2026Fast MoE Inference via Predictive Prefetching and Expert Replication
  6. 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
  7. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
  8. 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference