# Non-asymptotic model selection in block-diagonal mixture of polynomial experts models

arXiv: https://arxiv.org/abs/2104.08959 · HTML: https://mlsanity.com/paper/2104.08959/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 64 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 8 of 8)

- [Convergence Rates for Gaussian Mixtures of Experts](https://mlsanity.com/paper/1907.04377/) (2019)
- [A Universal Approximation Theorem for Mixture of Experts Models](https://mlsanity.com/paper/1602.03683/) (2016)
- [High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables](https://mlsanity.com/paper/1308.2302/) (2013)
- [Block-diagonal covariance selection for high-dimensional Gaussian graphical models](https://mlsanity.com/paper/1511.04033/) (2015)
- [Regularized Estimation and Feature Selection in Mixtures of Gaussian-Gated Experts Models](https://mlsanity.com/paper/1909.05494/) (2019)
- [Hyper-Spectral Image Analysis with Partially-Latent Regression and Spatial Markov Dependencies](https://mlsanity.com/paper/1409.8500/) (2014)
- [Nonlinear network-based quantitative trait prediction from transcriptomic data](https://mlsanity.com/paper/1701.07899/) (2017)
- [Rejoinder on: Minimal penalties and the slope heuristics: a survey](https://mlsanity.com/paper/1909.13499/) (2019)

## Built on by (most cited first, 5 of 5)

- [Statistical Advantages of Perturbing Cosine Router in Mixture of Experts](https://mlsanity.com/paper/2405.14131/) (2024)
- [Model Selection for Gaussian-gated Gaussian Mixture of Experts Using Dendrograms of Mixing Measures](https://mlsanity.com/paper/2505.13052/) (2025)
- [Functional Mixtures-of-Experts](https://mlsanity.com/paper/2202.02249/) (2022)
- [Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps](https://mlsanity.com/paper/2510.12744/) (2025)
- [Fast Model Selection and Stable Optimization for Softmax-Gated Multinomial-Logistic Mixture of Experts Models](https://mlsanity.com/paper/2602.07997/) (2026)

## Abstract (arXiv)

Model selection, via penalized likelihood type criteria, is a standard task in many statistical inference and machine learning problems. Progress has led to deriving criteria with asymptotic consistency results and an increasing emphasis on introducing non-asymptotic criteria. We focus on the problem of modeling non-linear relationships in regression data with potential hidden graph-structured interactions between the high-dimensional predictors, within the mixture of experts modeling framework. In order to deal with such a complex situation, we investigate a block-diagonal localized mixture of polynomial experts (BLoMPE) regression model, which is constructed upon an inverse regression and block-diagonal structures of the Gaussian expert covariance matrices. We introduce a penalized maximum likelihood selection criterion to estimate the unknown conditional density of the regression model. This model selection criterion allows us to handle the challenging problem of inferring the number of mixture components, the degree of polynomial mean functions, and the hidden block-diagonal structures of the covariance matrices, which reduces the number of parameters to be estimated and leads to a trade-off between complexity and sparsity in the model. In particular, we provide a strong theoretical guarantee: a finite-sample oracle inequality satisfied by the penalized maximum likelihood estimator with a Jensen-Kullback-Leibler type loss, to support the introduced non-asymptotic model selection criterion. The penalty shape of this criterion depends on the complexity of the considered random subcollection of BLoMPE models, including the relevant graph structures, the degree of polynomial mean functions, and the number of mixture components.