# An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling

arXiv: https://arxiv.org/abs/1707.03538 · HTML: https://mlsanity.com/paper/1707.03538/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 9 of its 91 references that are in the map.
- First posted on arXiv 12 Jul 2017.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 9 of 9)

- [Opponent Modeling in Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05559/) (2016)
- [Generative Image Modeling Using Spatial LSTMs](https://mlsanity.com/paper/1506.03478/) (2015)
- [A Universal Approximation Theorem for Mixture of Experts Models](https://mlsanity.com/paper/1602.03683/) (2016)
- [High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables](https://mlsanity.com/paper/1308.2302/) (2013)
- [Time series modeling by a regression approach based on a latent process](https://mlsanity.com/paper/1312.6969/) (2013)
- [Robust mixture of experts modeling using the t distribution](https://mlsanity.com/paper/1701.07429/) (2016)
- [A hidden process regression model for functional data description. Application to curve discrimination](https://mlsanity.com/paper/1312.6968/) (2013)
- [An Introduction to MM Algorithms for Machine Learning and Statistical](https://mlsanity.com/paper/1611.03969/) (2016)
- [Robust mixture of experts modeling using the skew t distribution](https://mlsanity.com/paper/1612.06879/) (2016)

## Built on by (most cited first, 1 of 1)

- [Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models](https://mlsanity.com/paper/1810.12161/) (2018)

## Abstract (arXiv)

Mixture-of-experts (MoE) models are a powerful paradigm for modeling of data arising from complex data generating processes (DGPs). In this article, we demonstrate how different MoE models can be constructed to approximate the underlying DGPs of arbitrary types of data. Due to the probabilistic nature of MoE models, we propose the maximum quasi-likelihood (MQL) estimator as a method for estimating MoE model parameters from data, and we provide conditions under which MQL estimators are consistent and asymptotically normal. The blockwise minorization-maximizatoin (blockwise-MM) algorithm framework is proposed as an all-purpose method for constructing algorithms for obtaining MQL estimators. An example derivation of a blockwise-MM algorithm is provided. We then present a method for constructing information criteria for estimating the number of components in MoE models and provide justification for the classic Bayesian information criterion (BIC). We explain how MoE models can be used to conduct classification, clustering, and regression and we illustrate these applications via a pair of worked examples.