# A similarity-based Bayesian mixture-of-experts model

arXiv: https://arxiv.org/abs/2012.02130 · HTML: https://mlsanity.com/paper/2012.02130/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 6 of its 54 references that are in the map.
- First posted on arXiv 3 Dec 2020.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 6 of 6)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [Automatic Differentiation Variational Inference](https://mlsanity.com/paper/1603.00788/) (2016)
- [Gaussian Parsimonious Clustering Models with Covariates and a Noise Component](https://mlsanity.com/paper/1711.05632/) (2017)

## Built on by (most cited first, 3 of 3)

- [Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts](https://mlsanity.com/paper/2601.03577/) (2026)
- [Probabilistic feature extraction, dose statistic prediction and dose mimicking for automated radiation therapy treatment planning](https://mlsanity.com/paper/2102.12569/) (2021)
- [CoCoAFusE: Beyond Mixtures of Experts via Model Fusion](https://mlsanity.com/paper/2505.01105/) (2025)

## Abstract (arXiv)

We present a new nonparametric mixture-of-experts model for multivariate regression problems, inspired by the probabilistic k-nearest neighbors algorithm. Using a conditionally specified model, predictions for out-of-sample inputs are based on similarities to each observed data point, yielding predictive distributions represented by Gaussian mixtures. Posterior inference is performed on the parameters of the mixture components as well as the distance metric using a mean-field variational Bayes algorithm accompanied with a stochastic gradient-based optimization procedure. The proposed method is especially advantageous in settings where inputs are of relatively high dimension in comparison to the data size, where input-output relationships are complex, and where predictive distributions may be skewed or multimodal. Computational studies on five datasets, of which two are synthetically generated, illustrate clear advantages of our mixture-of-experts method for high-dimensional inputs, outperforming competitor models both in terms of validation metrics and visual inspection.