# A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models

arXiv: https://arxiv.org/abs/2104.02640 · HTML: https://mlsanity.com/paper/2104.02640/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 8 of its 102 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 8 of 8)

- [Convergence Rates for Gaussian Mixtures of Experts](https://mlsanity.com/paper/1907.04377/) (2019)
- [A Universal Approximation Theorem for Mixture of Experts Models](https://mlsanity.com/paper/1602.03683/) (2016)
- [High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables](https://mlsanity.com/paper/1308.2302/) (2013)
- [Block-diagonal covariance selection for high-dimensional Gaussian graphical models](https://mlsanity.com/paper/1511.04033/) (2015)
- [A hidden process regression model for functional data description. Application to curve discrimination](https://mlsanity.com/paper/1312.6968/) (2013)
- [Nonlinear network-based quantitative trait prediction from transcriptomic data](https://mlsanity.com/paper/1701.07899/) (2017)
- [Rejoinder on: Minimal penalties and the slope heuristics: a survey](https://mlsanity.com/paper/1909.13499/) (2019)
- [Prediction regions through Inverse Regression](https://mlsanity.com/paper/1807.03184/) (2018)

## Built on by (most cited first, 7 of 7)

- [CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition](https://mlsanity.com/paper/2402.02526/) (2024)
- [Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts](https://mlsanity.com/paper/2305.07572/) (2023)
- [Risk Bounds for Mixture Density Estimation on Compact Domains via the h-Lifted Kullback--Leibler Divergence](https://mlsanity.com/paper/2404.12586/) (2024)
- [Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps](https://mlsanity.com/paper/2510.12744/) (2025)
- [Mixture of partially linear experts](https://mlsanity.com/paper/2405.02905/) (2024)
- [Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts](https://mlsanity.com/paper/2601.19811/) (2026)
- [Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models](https://mlsanity.com/paper/2602.07997/) (2026)

## Abstract (arXiv)

Mixture of experts (MoE) are a popular class of statistical and machine learning models that have gained attention over the years due to their flexibility and efficiency. In this work, we consider Gaussian-gated localized MoE (GLoME) and block-diagonal covariance localized MoE (BLoME) regression models to present nonlinear relationships in heterogeneous data with potential hidden graph-structured interactions between high-dimensional predictors. These models pose difficult statistical estimation and model selection questions, both from a computational and theoretical perspective. This paper is devoted to the study of the problem of model selection among a collection of GLoME or BLoME models characterized by the number of mixture components, the complexity of Gaussian mean experts, and the hidden block-diagonal structures of the covariance matrices, in a penalized maximum likelihood estimation framework. In particular, we establish non-asymptotic risk bounds that take the form of weak oracle inequalities, provided that lower bounds for the penalties hold. The good empirical behavior of our models is then demonstrated on synthetic and real datasets.