mlsanityData to 7 Oct 2026

Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of experts

arXiv 2009.10622 ↗First posted 22 Sep 2020math.ST cs.AI cs.LGcited by 6

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

We investigate the estimation properties of the mixture of experts (MoE) model in a high-dimensional setting, where the number of predictors is much larger than the sample size, and for which the literature is particularly lacking in theoretical results. We consider the class of softmax-gated Gaussian MoE (SGMoE) models, defined as MoE models with softmax gating functions and Gaussian experts, and focus on the theoretical properties of their l1-regularized estimation via the Lasso. To the best of our knowledge, we are the first to investigate the l1-regularization properties of SGMoE models from a non-asymptotic perspective, under the mildest assumptions, namely the boundedness of the parameter space. We provide a lower bound on the regularization parameter of the Lasso penalty that ensures non-asymptotic theoretical control of the Kullback--Leibler loss of the Lasso estimator for SGMoE models. Finally, we carry out a simulation study to empirically validate our theoretical findings.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 10 of its 93 references that are in the map. First posted on arXiv 22 Sep 2020.

Builds on

10 of 93 references.

Closest ancestors · 3

Same direction, newest first

  1. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts14 Jul
  2. 2019Convergence Rates for Gaussian Mixtures of Experts9 Jul
  3. 2018Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models29 Oct

Other references · 7

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2013Deep Learning of Representations: Looking Forwardcited by 84
  7. 2016A Universal Approximation Theorem for Mixture of Experts Modelscited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 4; 2026: 1.

6 papers.

  1. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17
  2. 2025Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measurescited by 3
  3. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweepscited by 2
  4. 2025CompeteSMoE -- Statistically Guaranteed Mixture of Experts Training via Competitioncited by 1
  5. 2025Scalable Subset Selection in Linear Mixed Modelscited by 1
  6. 2026Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models
  2. 2019Regularized Estimation and Feature Selection in Mixtures of Gaussian-Gated Experts Models
  3. 2024On Least Square Estimation in Softmax Gating Mixture of Experts
  4. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
  5. 2025Convergence Rates for Softmax Gating Mixture of Experts
  6. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Experts
  7. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models
  8. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Experts