mlsanityData to 7 Oct 2026

Turn Waste into Worth: Rectifying Top-k Router of MoE

arXiv 2402.12399 ↗First posted 17 Feb 2024cs.LG cs.AI cs.CLcited by 11

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparse Mixture of Experts (MoE) models are popular for training large language models due to their computational efficiency. However, the commonly used top-k routing mechanism suffers from redundancy computation and memory costs due to the unbalanced routing. Some experts are overflow, where the exceeding tokens are dropped. While some experts are vacant, which are padded with zeros, negatively impacting model performance. To address the dropped tokens and padding, we propose the Rectify-Router, comprising the Intra-GPU Rectification and the Fill-in Rectification. The Intra-GPU Rectification handles dropped tokens, efficiently routing them to experts within the GPU where they are located to avoid inter-GPU communication. The Fill-in Rectification addresses padding by replacing padding tokens with the tokens that have high routing scores. Our experimental results demonstrate that the Intra-GPU Rectification and the Fill-in Rectification effectively handle dropped tokens and padding, respectively. Furthermore, the combination of them achieves superior performance, surpassing the accuracy of the vanilla top-1 router by 4.7%.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 23 of its 31 references that are in the map. First posted on arXiv 17 Feb 2024.

Builds on

23 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 21

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  10. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  11. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  12. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
9 more
  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  2. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  3. 2023From Sparse to Soft Mixtures of Expertscited by 245
  4. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  5. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  6. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  7. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  8. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  9. 2022Efficient Language Modeling with Sparse all-MLPcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 2.

11 papers.

  1. 2024MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Expertscited by 144
  2. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  3. 2025SonicMoE: Accelerating MoE with IO and Tile-aware Optimizationscited by 20
  4. 2024WDMoE: Wireless Distributed Large Language Models with Mixture of Expertscited by 18
  5. 2025CLIP-UP: A Simple and Efficient Mixture-of-Experts CLIP Training Recipe with Sparse Upcyclingcited by 5
  6. 2025Decoding Knowledge Attribution in Mixture-of-Experts: A Framework of Basic-Refinement Collaboration and Efficiency Analysiscited by 5
  7. 2024Beyond Parameter Count: Implicit Bias in Soft Mixture of Expertscited by 1
  8. 2025Model Hemorrhage and the Robustness Limits of Large Language Modelscited by 1
  9. 2025TokenSmith: Streamlining Data Editing, Search, and Inspection for Large-Scale Language Model Training and Interpretabilitycited by 0
  10. 2026Aspect-Based Sentiment Analysis for Future Tourism Experiences: A BERT-MoE Framework for Persian User Reviewscited by 0
  11. 2026Decoding the Skew: Distribution-Aware MoE Inference with Adaptive Kernel Dispatchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routing
  2. 2025Maximum Score Routing For Mixture-of-Experts
  3. 2025MoETuner: Optimized Mixture of Expert Serving with Balanced Expert Placement and Token Routing
  4. 2024ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
  5. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing
  6. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  7. 2026MoRA: MoE Pruning via Router Bias Learning and Expert Approximation
  8. 2026Expert Coupling in MoE Pretraining: Reducing All-to-All Overhead with Correlated Placement and Token Shuffling