mlsanityData to 7 Oct 2026

ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing

arXiv 2412.14711 ↗First posted 19 Dec 2024cs.LGcited by 53

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Sparsely activated Mixture-of-Experts (MoE) models are widely adopted to scale up model capacity without increasing the computation budget. However, vanilla TopK routers are trained in a discontinuous, non-differentiable way, limiting their performance and scalability. To address this issue, we propose ReMoE, a fully differentiable MoE architecture that offers a simple yet effective drop-in replacement for the conventional TopK+Softmax routing, utilizing ReLU as the router instead. We further propose methods to regulate the router's sparsity while balancing the load among experts. ReMoE's continuous nature enables efficient dynamic allocation of computation across tokens and layers, while also exhibiting domain specialization. Our experiments demonstrate that ReMoE consistently outperforms vanilla TopK-routed MoE across various model sizes, expert counts, and levels of granularity. Furthermore, ReMoE exhibits superior scalability with respect to the number of experts, surpassing traditional MoE architectures. The implementation based on Megatron-LM is available at https://github.com/thu-ml/ReMoE.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 57 of its 63 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

57 of 63 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Ultra-Sparse Memory Network19 Nov
  2. 2023SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention13 Dec
  3. 2023Soft Merging of Experts with Adaptive Routing6 Jun
  4. 2022Mixture of Attention Heads: Selecting Attention Heads Per Token11 Oct
  5. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  6. 2021Taming Sparsely Activated Transformer with Stochastic Experts8 Oct
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 50

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
38 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2024Mixtral of Expertscited by 1,785
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019Root Mean Square Layer Normalizationcited by 1,425
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  11. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  12. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  13. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  14. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  15. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  16. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  17. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  18. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  19. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  20. 2023From Sparse to Soft Mixtures of Expertscited by 245
  21. 2021Hash Layers For Large Sparse Modelscited by 236
  22. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  23. 2024Mixture of LoRA Expertscited by 179
  24. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  25. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
  26. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuningcited by 148
  27. 2024Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Modelscited by 140
  28. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  29. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 121
  30. 2015Gradient Estimation Using Stochastic Computation Graphscited by 120
  31. 2019Large Memory Layers with Product Keyscited by 120
  32. 2024Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costscited by 51
  33. 2024ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Modelscited by 43
  34. 2024Memory Layers at Scalecited by 39
  35. 2024Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-trainingcited by 37
  36. 2024GRIN: GRadient-INformed MoEcited by 16
  37. 2024Scattered Mixture-of-Experts Implementationcited by 15
  38. 2024RoDE: Linear Rectified Mixture of Diverse Experts for Food Large Multi-Modal Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 23; 2026: 30.

53 papers.

Sort:

By area:

  1. 2025Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Expertscited by 13
  2. 2025PVChat: Personalized Video Chat with One-Shot Learningcited by 13
  3. 2025LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Expertscited by 12
  4. 2025Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Expertscited by 10
  5. 2025HiMoE-VLA: Hierarchical Mixture-of-Experts for Generalist Vision-Language-Action Policiescited by 10
  6. 2025Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Expertscited by 9
  7. 2025Dense Backpropagation Improves Training for Sparse Mixture-of-Expertscited by 8
  8. 2025Learning to Skip the Middle Layers of Transformerscited by 8
  9. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Modelscited by 8
  10. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysiscited by 7
  11. 2025BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsitycited by 6
  12. 2025Understanding the Mixture-of-Experts with Nadaraya-Watson Kernelcited by 6
38 more
  1. 2025Towards Stable and Effective Reinforcement Learning for Mixture-of-Expertscited by 6
  2. 2025Spark Transformer: Reactivating Sparsity in FFN and Attentioncited by 5
  3. 2025Route Experts by Sequence, not by Tokencited by 5
  4. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specializationcited by 5
  5. 2026Grouter: Decoupling Routing from Representation for Accelerated MoE Trainingcited by 5
  6. 2025Robust Experts: the Effect of Adversarial Training on CNNs with Sparse Mixture-of-Experts Layerscited by 4
  7. 2026φ-Balancing for Mixture-of-Experts Trainingcited by 4
  8. 2025Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gatecited by 3
  9. 2025Maximum Score Routing For Mixture-of-Expertscited by 3
  10. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  11. 2026Expert Threshold Routing for Autoregressive Language Modeling with Dynamic Computation Allocation and Load Balancingcited by 3
  12. 2026Expert-Choice Routing Enables Adaptive Computation in Diffusion Language Modelscited by 3
  13. 2025DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-trainingcited by 2
  14. 2026Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activationscited by 2
  15. 2026Multi-Head LatentMoE and Head Parallel: Communication-Efficient and Deterministic MoE Parallelismcited by 2
  16. 2026Routing Absorption in Sparse Attention: Why Random Gates Are Hard to Beatcited by 2
  17. 2026Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Modelscited by 2
  18. 2025Rethinking Sparse Mixture of Experts from a Unified Perspectivecited by 1
  19. 2025VersatileFFN: Achieving Parameter Efficiency in LLMs via Adaptive Wide-and-Deep Reusecited by 1
  20. 2026DirMoE: Dirichlet-routed Mixture of Expertscited by 1
  21. 2026Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Qualitycited by 1
  22. 2026UniPool: A Globally Shared Expert Pool for Mixture-of-Expertscited by 1
  23. 2025Condition Weaving Meets Expert Modulation: Towards Universal and Controllable Image Generationcited by 0
  24. 2025Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Modelscited by 0
  25. 2026Multi-Domain Learning with Global Expert Mappingcited by 0
  26. 2026Teacher-Guided Routing for Sparse Vision Mixture-of-Expertscited by 0
  27. 2026DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devicescited by 0
  28. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
  29. 2026ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Expertscited by 0
  30. 2026STAR: Rethinking MoE Routing as Structure-Aware Subspace Learningcited by 0
  31. 2026GeMoE: Gating Entropy is All You Need for Uncertainty-aware Adaptive Routing in MoE-based Large Vision-Language Modelscited by 0
  32. 2026Focusing on What Matters: Saliency-Harnessing Accurate Routing for Diffusion MoEcited by 0
  33. 2026Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligencecited by 0
  34. 2026SpecDrop: Parameter-Free Category-Conditioned Routing for Modular Specializationcited by 0
  35. 2026PRIME: Mitigating Subgroup Optimization Competition in Shared CTR Top Networks with Plug-in Residual Input-Conditioned Mixture of Expertcited by 0
  36. 2026Beyond Magnitude: Contrastive Routing for Modular Mixture-of-Expertscited by 0
  37. 2026ExpertHTR: Unified Handwritten Text Recognition with Multi-Task Learning and Sparse Mixture-of-Expertscited by 0
  38. 2026IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Expertscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ReMoE: Boosting Expert Reuse through Router Fine-Tuning in Memory-Constrained MoE LLM Inference
  2. 2025ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization
  3. 2025Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
  4. 2026SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs
  5. 2024Layerwise Recurrent Router for Mixture-of-Experts
  6. 2026Hierarchical Mixture-of-Experts with Two-Stage Optimization
  7. 2026MoE Routing Testbed: Studying Expert Specialization and Routing Behavior at Small Scale
  8. 2026Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation