# Gaussian Process-Gated Hierarchical Mixtures of Experts

arXiv: https://arxiv.org/abs/2302.04947 · HTML: https://mlsanity.com/paper/2302.04947/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 15 of its 60 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 15 of 15)

- [Deep Neural Networks as Gaussian Processes](https://mlsanity.com/paper/1711.00165/) (2017)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Distilling a Neural Network Into a Soft Decision Tree](https://mlsanity.com/paper/1711.09784/) (2017)
- [Scalable Variational Gaussian Process Classification](https://mlsanity.com/paper/1411.2005/) (2014)
- [Stochastic Variational Deep Kernel Learning](https://mlsanity.com/paper/1611.00336/) (2016)
- [Hilbert Space Methods for Reduced-Rank Gaussian Process Regression](https://mlsanity.com/paper/1401.5508/) (2014)
- [Random Feature Expansions for Deep Gaussian Processes](https://mlsanity.com/paper/1610.04386/) (2016)
- [GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning](https://mlsanity.com/paper/2102.07868/) (2021)
- [Deep Neural Networks as Point Estimates for Deep Gaussian Processes](https://mlsanity.com/paper/2105.04504/) (2021)
- [The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective](https://mlsanity.com/paper/2106.06529/) (2021)
- [Amortized Variational Inference for Simple Hierarchical Models](https://mlsanity.com/paper/2111.03144/) (2021)
- [Hierarchical Mixture-of-Experts Model for Large-Scale Gaussian Process Regression](https://mlsanity.com/paper/1412.3078/) (2014)
- [Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs](https://mlsanity.com/paper/1503.02424/) (2015)
- [GP-BART: a novel Bayesian additive regression trees approach using Gaussian processes](https://mlsanity.com/paper/2204.02112/) (2022)
- [AutoGP: Exploring the Capabilities and Limitations of Gaussian Process Models](https://mlsanity.com/paper/1610.05392/) (2016)

## Built on by (most cited first, 2 of 2)

- [Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Prediction](https://mlsanity.com/paper/2509.12227/) (2025)
- [R^2-HGP: A Double-Regularized Gaussian Process for Heterogeneous Transfer Learning](https://mlsanity.com/paper/2512.10258/) (2025)

## Abstract (arXiv)

In this paper, we propose novel Gaussian process-gated hierarchical mixtures of experts (GPHMEs). Unlike other mixtures of experts with gating models linear in the input, our model employs gating functions built with Gaussian processes (GPs). These processes are based on random features that are non-linear functions of the inputs. Furthermore, the experts in our model are also constructed with GPs. The optimization of the GPHMEs is performed by variational inference. The proposed GPHMEs have several advantages. They outperform tree-based HME benchmarks that partition the data in the input space, and they achieve good performance with reduced complexity. Another advantage is the interpretability they provide for deep GPs, and more generally, for deep Bayesian neural networks. Our GPHMEs demonstrate excellent performance for large-scale data sets, even with quite modest sizes.