mlsanityData to 7 Oct 2026

Convergence Rates for Gaussian Mixtures of Experts

arXiv 1907.04377 ↗First posted 9 Jul 2019math.ST cs.LG stat.MLcited by 39

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

We provide a theoretical treatment of over-specified Gaussian mixtures of experts with covariate-free gating networks. We establish the convergence rates of the maximum likelihood estimation (MLE) for these models. Our proof technique is based on a novel notion of algebraic independence of the expert functions. Drawing on optimal transport theory, we establish a connection between the algebraic independence and a certain class of partial differential equations (PDEs). Exploiting this connection allows us to derive convergence rates and minimax lower bounds for parameter estimation.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 5 of its 30 references that are in the map. First posted on arXiv 9 Jul 2019.

Builds on

5 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Learning in Gated Neural Networks6 Jun
  2. 2018Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms21 Feb

Other references · 3

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2013Learning Factored Representations in a Deep Mixture of Expertscited by 275
  3. 2016Singularity structures and impacts on parameter estimation in finite mixtures of distributionscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 3; 2022: 1; 2023: 8; 2024: 10; 2025: 7; 2026: 9.

39 papers.

Sort:

By area:

  1. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 67
  2. 2024Mixture of Experts Meets Prompt-Based Continual Learningcited by 43
  3. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 33
  4. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  5. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  6. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  7. 2024On Least Square Estimation in Softmax Gating Mixture of Expertscited by 23
  8. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  9. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17
  10. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  11. 2024Is Temperature Sample Efficient for Softmax Gaussian Mixture of Experts?cited by 15
  12. 2024Revisiting Prefix-tuning: Statistical Benefits of Reparameterization among Promptscited by 15
27 more
  1. 2021A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts modelscited by 7
  2. 2020Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of expertscited by 6
  3. 2022Strong identifiability and parameter learning in regression with heterogeneous responsecited by 6
  4. 2021Non-asymptotic model selection in block-diagonal mixture of polynomial experts modelscited by 5
  5. 2021Uniform Consistency in Nonparametric Mixture Modelscited by 5
  6. 2025Revisit Visual Prompt Tuning: The Expressiveness of Prompt Expertscited by 5
  7. 2023SQ Lower Bounds for Learning Mixtures of Linear Classifierscited by 4
  8. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  9. 2023Mixed Regression via Approximate Message Passingcited by 3
  10. 2025Sigmoid Self-Attention has Lower Sample Complexity than Softmax Self-Attention: A Mixture-of-Experts Perspectivecited by 3
  11. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  12. 2024Approximation Rates and VC-Dimension Bounds for (P)ReLU MLP Mixture of Expertscited by 2
  13. 2024On Parameter Estimation in Deviated Gaussian Mixture of Expertscited by 2
  14. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepscited by 2
  15. 2023Joint Problems in Learning Multiple Dynamical Systemscited by 1
  16. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
  17. 2025Limitations of refinement methods for weak to strong generalizationcited by 1
  18. 2026Robustness of Mixtures of Experts to Feature Noisecited by 1
  19. 2026A Statistical Theory of Gated Attention through the Lens of Hierarchical Mixture of Expertscited by 1
  20. 2026Soft-to-Hard Routing in Sparse Mixture-of-Experts Modelscited by 1
  21. 2023Quantum-AI empowered Intelligent Surveillance: Advancing Public Safety Through Innovative Contraband Detectioncited by 0
  22. 2026Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Expertscited by 0
  23. 2026Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Modelscited by 0
  24. 2026From Isolation to Integration: Building an Adaptive Expert Forest for Pre-Trained Model-based Class-Incremental Learningcited by 0
  25. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  26. 2026FedSPM: Routing-Enabled Federated Learning under Dual Heterogeneity via Semiparametric Mixturecited by 0
  27. 2026Characterizing Heterogeneous Rates in Finite Mixture Estimation via Partial Optimal Transportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts
  2. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  3. 2019Regularized Estimation and Feature Selection in Mixtures of Gaussian-Gated Experts Models
  4. 2025Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures
  5. 2022Functional mixture-of-experts for classification
  6. 2024Learning Mixtures of Gaussians Using Diffusion Models
  7. 2026Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs
  8. 2024Understanding Expert Structures on Minimax Parameter Estimation in Contaminated Mixture of Experts