mlsanityData to 14 Oct 2026

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts

arXiv 2504.12463 ↗First posted 16 Apr 2025cs.LG cs.AIcited by 10

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

Mixture of Experts (MoE) pretraining is more scalable than dense Transformer pretraining, because MoEs learn to route inputs to a sparse set of their feedforward parameters. However, this means that MoEs only receive a sparse backward update, leading to training instability and suboptimal performance. We present a lightweight approximation method that gives the MoE router a dense gradient update while continuing to sparsely activate its parameters. Our method, which we refer to as Default MoE, substitutes missing expert activations with default outputs consisting of an exponential moving average of expert outputs previously seen over the course of training. This allows the router to receive signals from every expert for each token, leading to significant improvements in training performance. Our Default MoE outperforms standard TopK routing in a variety of settings without requiring significant computational overhead. Code: https://github.com/vatsal0/default-moe.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 22 of its 40 references that are in the map. First posted on arXiv 16 Apr 2025.

Builds on

22 of 40 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing19 Dec
  2. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts28 Aug
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 18

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2016Layer Normalizationcited by 5,669
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  9. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  10. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  12. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
6 more
  1. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 178
  2. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  3. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  4. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 90
  5. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 54
  6. 2023Sparse Backpropagation for MoE Trainingcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2025Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMscited by 7
  2. 2026Transformer as an Euler Discretization of Score-based Variational Flowcited by 0
  3. 2026Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Executioncited by 0
  4. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
  5. 2026TEXAS: Task-Expert-Aware Supervision for Downstream Mixture-of-Experts LLM Adaptationcited by 0
  6. 2026Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Expertscited by 0
  7. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  8. 2026Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Expertscited by 0
  9. 2026Dense Mixture-of-Experts as a Reparameterized Wide FFN: A Granularity Sweep at Fixed Computecited by 0
  10. 2026Principled Top-k Selection for Language Models with Hybrid Gradientscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Teacher-Guided Routing for Sparse Vision Mixture-of-Experts
  2. 2026Dense vs Sparse Pretraining at Tiny Scale: Active-Parameter vs Total-Parameter Matching
  3. 2025Load Balancing Mixture of Experts with Similarity Preserving Routers
  4. 2025Continual Pre-training of MoEs: How robust is your router?
  5. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  6. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  7. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
  8. 2026Synergistic Intra- and Cross-Layer Regularization Losses for MoE Expert Specialization