mlsanityData to 7 Oct 2026

An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling

arXiv 1707.03538 ↗First posted 12 Jul 2017stat.ML cs.LGcited by 1

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Mixture-of-experts (MoE) models are a powerful paradigm for modeling of data arising from complex data generating processes (DGPs). In this article, we demonstrate how different MoE models can be constructed to approximate the underlying DGPs of arbitrary types of data. Due to the probabilistic nature of MoE models, we propose the maximum quasi-likelihood (MQL) estimator as a method for estimating MoE model parameters from data, and we provide conditions under which MQL estimators are consistent and asymptotically normal. The blockwise minorization-maximizatoin (blockwise-MM) algorithm framework is proposed as an all-purpose method for constructing algorithms for obtaining MQL estimators. An example derivation of a blockwise-MM algorithm is provided. We then present a method for constructing information criteria for estimating the number of components in MoE models and provide justification for the classic Bayesian information criterion (BIC). We explain how MoE models can be used to conduct classification, clustering, and regression and we illustrate these applications via a pair of worked examples.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 9 of its 91 references that are in the map. First posted on arXiv 12 Jul 2017.

Builds on

9 of 91 references.

References · 9

  1. 2016Opponent Modeling in Deep Reinforcement Learningcited by 94
  2. 2015Generative Image Modeling Using Spatial LSTMscited by 35
  3. 2016A Universal Approximation Theorem for Mixture of Experts Modelscited by 17
  4. 2013High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variablescited by 9
  5. 2013Time series modeling by a regression approach based on a latent processcited by 6
  6. 2016Robust mixture of experts modeling using the t distributioncited by 4
  7. 2013A hidden process regression model for functional data description. Application to curve discriminationcited by 3
  8. 2016An Introduction to MM Algorithms for Machine Learning and Statisticalcited by 1
  9. 2016Robust mixture of experts modeling using the skew t distributioncited by 1

Built on by

Cited by year

Citing papers per year: 2018: 1.

1 paper.

  1. 2018Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Modelscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications
  2. 2024On Least Square Estimation in Softmax Gating Mixture of Experts
  3. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts
  4. 2021A non-asymptotic approach for model selection via penalization in high-dimensional mixture of experts models
  5. 2026Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneity
  6. 2025Mixture of Experts (MoE): A Big Data Perspective
  7. 2024Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study
  8. 2025Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective