Sticky Routing: Training MoE Models for Memory-Efficient Inference
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices. Existing remedies are either system-level (caching heuristics) or post-hoc (router fine-tuning), leaving the root cause unchanged during pretraining. We propose StickyMoE, a differentiable routing consistency loss that penalises abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. StickyMoE requires no architectural changes, adds a single hyperparameter lambda, and unlike post-hoc methods, allows expert representations and routing decisions to co-adapt from the first training step. Experiments on small-scale MoE language models show that StickyMoE reduces the expert switch rate by up to 60% with less than 4% perplexity degradation, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. Routing temporal locality is most efficiently instilled at training time.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 16 of its 23 references that are in the map. First posted on arXiv 12 Jun 2026.
Builds on
16 of 23 references.
Closest ancestors · 3
Same direction, newest first
- 2025Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models21 May
- 2022Mixture-of-Experts with Expert Choice Routing18 Feb
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan
Other references · 13
- 2017Neural Discrete Representation Learningcited by 4,650
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
- 2016Using the Output Embedding to Improve Language Modelscited by 266
- 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
- 2021Hash Layers For Large Sparse Modelscited by 236
- 2023Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inferencecited by 101
- 2024Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Modelscited by 69
- 2024MoE-Infinity: Efficient MoE Inference on Personal Machines with Sparsity-Aware Expert Cachecited by 48
- 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inferencecited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Elbow-Based MoE Routing: A Training-Free Inference Time Plugin for Expert Selection
- 2026Cache-Aware Joint Router Adaptation for Memory-Efficient MoE Inference
- 2026Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
- 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
- 2026MawForge: Memory-Bounded Expert Materialization for Local Mixture-of-Experts Inference
- 2026SpecPrefetch: Parameter-Efficient Expert Prefetching for Sparse MoE Foundation Models
- 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
- 2026Self-Routing: Parameter-Free Expert Routing from Hidden States