mlsanityData to 7 Oct 2026

MoE theory and gating functions

188 papers0 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: experts, moe, mixture, mixture experts, gating

Key facts. MoE theory and gating functions: 188 papers on arXiv since 2017 (data through 7 Oct 2026). 23 of its 188 mapped papers (12%) cite “Convergence Rates for Gaussian Mixtures of Experts” (2019, arXiv 1907.04377). 20 of its 188 mapped papers (11%) cite “Demystifying Softmax Gating Function in Gaussian Mixture of Experts” (2023, arXiv 2305.03288). 6 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

6 new papers, 17 Sep – 7 Oct; 0 this week.

  1. Earlier · 6
  2. Localized Operator Learning with Adaptive Partition-of-Unity Mixture-of-Expert Networks also fits: Vision mixture-of-experts
  3. Mixture-of-Experts for Cryptocurrency Order Execution: Training Stability, Tail Risk, and Failure Modes also fits: MoE in reinforcement learning and robotics
  4. SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining
  5. Redundancy Meets Synergy: Dependency-aware Expert Selection for MoE via Submodular Optimization
  6. Scaling Laws for Looped Mixture of Experts also fits: Expert routing strategies
  7. E-MoE: Enhanced Mixture-of-Experts for Non-Factorized Diffusion Language Models also fits: MoE large language models

Foundations

Most cited by papers here

  1. Convergence Rates for Gaussian Mixtures of Experts2019cited by 23 here
  2. Demystifying Softmax Gating Function in Gaussian Mixture of Experts2023cited by 20 here
  3. DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning2021cited by 19 here
  4. A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts2023cited by 17 here
  5. Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts2023cited by 13 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 90 here
  2. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 51 here
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 48 here
  4. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 18 here
  5. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 11 here
  6. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 20 here
  7. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 13 here
  8. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 12 here
  9. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 17 here
  10. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 11 here
14 more it builds on, not in this chain
  1. 2019Convergence Rates for Gaussian Mixtures of Expertscited by 23 here
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 77 here
  3. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 19 here
  4. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 10 here
  5. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 33 here
  6. 2022Mixture-of-Experts with Expert Choice Routingcited by 27 here
  7. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 14 here
  8. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 13 here
  9. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 15 here
  10. 2023From Sparse to Soft Mixtures of Expertscited by 17 here
  11. 2024Mixtral of Expertscited by 31 here
  12. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 34 here
  13. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 10 here
  14. 2024DeepSeek-V3 Technical Reportcited by 13 here

The 24 most cited of 337.

Browse

30 of 188, ranked 6–35

  1. 6Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Experts202312
  2. 7On Least Square Estimation in Softmax Gating Mixture of Experts20248
  3. 8Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts20247
  4. 9Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts20196
  5. 10A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications20255
  6. 11On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions20245
  7. 12A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models20215
  8. 13Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models20185
  9. 14Interpretable Mixture of Experts20224
  10. 15Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts20204
All 30
  1. 16Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms20184
  2. 17Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient20253
  3. 18Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures20253
  4. 19Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?20243
  5. 20Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning20233
  6. 21Functional mixture-of-experts for classification20223
  7. 22Mixture of experts models for multilevel data: modelling framework and approximation theory20223
  8. 23Dropout Regularization in Hierarchical Mixture of Experts20183
  9. 24How to Set the Learning Rate for Large-Scale Pre-training?20262
  10. 25On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks20252
  11. 26QMoE: A Quantum Mixture of Experts Framework for Scalable Quantum Neural Networks20252
  12. 27μ-Parametrization for Mixture of Experts20252
  13. 28Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps20252
  14. 29Mixture-of-Experts Operator Transformer for Large-Scale PDE Pre-Training20252
  15. 30On Parameter Estimation in Deviated Gaussian Mixture of Experts20242
  16. 31Improving Expert Specialization in Mixture of Experts20232
  17. 32Revisiting Single-gated Mixtures of Experts20232
  18. 33Task-Specific Expert Pruning for Sparse Mixture-of-Experts20222
  19. 34Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs20222
  20. 35Prediction Sets for High-Dimensional Mixture of Experts Models20222