mlsanityData to 7 Oct 2026

Gaussian Process-Gated Hierarchical Mixtures of Experts

arXiv 2302.04947 ↗First posted 9 Feb 2023cs.LG stat.MLcited by 2

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

In this paper, we propose novel Gaussian process-gated hierarchical mixtures of experts (GPHMEs). Unlike other mixtures of experts with gating models linear in the input, our model employs gating functions built with Gaussian processes (GPs). These processes are based on random features that are non-linear functions of the inputs. Furthermore, the experts in our model are also constructed with GPs. The optimization of the GPHMEs is performed by variational inference. The proposed GPHMEs have several advantages. They outperform tree-based HME benchmarks that partition the data in the input space, and they achieve good performance with reduced complexity. Another advantage is the interpretability they provide for deep GPs, and more generally, for deep Bayesian neural networks. Our GPHMEs demonstrate excellent performance for large-scale data sets, even with quite modest sizes.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 15 of its 60 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

15 of 60 references.

References · 15

  1. 2017Deep Neural Networks as Gaussian Processescited by 617
  2. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  3. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  4. 2014Scalable Variational Gaussian Process Classificationcited by 226
  5. 2016Stochastic Variational Deep Kernel Learningcited by 92
  6. 2014Hilbert Space Methods for Reduced-Rank Gaussian Process Regressioncited by 88
  7. 2016Random Feature Expansions for Deep Gaussian Processescited by 32
  8. 2021GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learningcited by 26
  9. 2021Deep Neural Networks as Point Estimates for Deep Gaussian Processescited by 26
  10. 2021The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspectivecited by 21
  11. 2021Amortized Variational Inference for Simple Hierarchical Modelscited by 19
  12. 2014Hierarchical Mixture-of-Experts Model for Large-Scale Gaussian Process Regressioncited by 15
3 more
  1. 2015Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputscited by 11
  2. 2022GP-BART: a novel Bayesian additive regression trees approach using Gaussian processescited by 7
  3. 2016AutoGP: Exploring the Capabilities and Limitations of Gaussian Process Modelscited by 6

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Learning to Route: Per-Sample Adaptive Routing for Multimodal Multitask Predictioncited by 0
  2. 2025R^2-HGP: A Double-Regularized Gaussian Process for Heterogeneous Transfer Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Fast Deep Mixtures of Gaussian Process Experts
  2. 2019Enriched Mixtures of Gaussian Process Experts
  3. 2018Learning Deep Mixtures of Gaussian Process Experts Using Sum-Product Networks
  4. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions
  5. 2025Dendrograms of Mixing Measures for Softmax-Gated Gaussian Mixture of Experts: Consistency Without Model Sweeps
  6. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Experts
  7. 2017Embarrassingly Parallel Inference for Gaussian Processes
  8. 2026On Bayesian Softmax-Gated Mixture-of-Experts Models