mlsanityData to 7 Oct 2026

Sparse Backpropagation for MoE Training

arXiv 2310.00811 ↗First posted 1 Oct 2023cs.LG cs.AI cs.CLcited by 16

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

One defining characteristic of Mixture-of-Expert (MoE) models is their capacity for conducting sparse computation via expert routing, leading to remarkable scalability. However, backpropagation, the cornerstone of deep learning, requires dense computation, thereby posting challenges in MoE gradient computations. Here, we introduce SparseMixer, a scalable gradient estimator that bridges the gap between backpropagation and sparse expert routing. Unlike typical MoE training which strategically neglects certain gradient terms for the sake of sparse computation and scalability, SparseMixer provides scalable gradient approximations for these terms, enabling reliable gradient estimation in MoE training. Grounded in a numerical ODE framework, SparseMixer harnesses the mid-point method, a second-order ODE solver, to deliver precise gradient approximations with negligible computational overhead. Applying SparseMixer to Switch Transformer on both pre-training and machine translation tasks, SparseMixer showcases considerable performance gain, accelerating training convergence up to 2 times.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 30 of its 43 references that are in the map. First posted on arXiv 1 Oct 2023.

Builds on

30 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts24 Sep

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  10. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
17 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  3. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  4. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  5. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 349
  6. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  7. 2020Understanding the Difficulty of Training Transformerscited by 210
  8. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  9. 2014A* Samplingcited by 150
  10. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
  11. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  12. 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 104
  13. 2017REBAR: Low-variance, unbiased gradient estimates for discrete latent variable modelscited by 78
  14. 2015MuProp: Unbiased Backpropagation for Stochastic Neural Networkscited by 36
  15. 2023Bridging Discrete and Backpropagation: Straight-Through and Beyondcited by 31
  16. 2022Gradient Estimation with Discrete Stein Operatorscited by 22
  17. 2022METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signalscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2024A Survey on Model MoErging: Recycling and Routing Among Specialized Experts for Collaborative Learningcited by 52
  3. 2026Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scalecited by 25
  4. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 24
  5. 2024GRIN: GRadient-INformed MoEcited by 16
  6. 2025Dense Backpropagation Improves Training for Sparse Mixture-of-Expertscited by 8
  7. 2024Mixture of Experts with Mixture of Precisions for Tuning Quality of Servicecited by 7
  8. 2025Continual Pre-training of MoEs: How robust is your router?cited by 5
  9. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projectioncited by 4
  10. 2024Advancing On-Device Neural Network Training with TinyPropv2: Dynamic, Sparse, and Efficient Backpropagationcited by 3
  11. 2026DirMoE: Dirichlet-routed Mixture of Expertscited by 1
  12. 2026PR2: Predictive Routing Replay for MoE-Based LLM Reinforcement Learningcited by 1
4 more
  1. 2025SlimMoE: Structured Compression of Large MoE Models via Expert Slimming and Distillationcited by 0
  2. 2026UB-SMoE: Universally Balanced Sparse Mixture-of-Experts for Resource-adaptive Federated Fine-tuning of Foundation Modelscited by 0
  3. 2026ProbMoE: Differentiable Probabilistic Routing for Mixture-of-Expertscited by 0
  4. 2026Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
  2. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  3. 2022Task-Specific Expert Pruning for Sparse Mixture-of-Experts
  4. 2025TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
  5. 2025DualSparse-MoE: Coordinating Tensor/Neuron-Level Sparsity with Expert Partition and Reconstruction
  6. 2024Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
  7. 2021Sparse MoEs meet Efficient Ensembles
  8. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study