mlsanityData to 7 Oct 2026

Mixture of experts models for multilevel data: modelling framework and approximation theory

arXiv 2209.15207 ↗First posted 30 Sep 2022math.ST cs.LG cs.NEcited by 3

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

Multilevel data are prevalent in many real-world applications. However, it remains an open research problem to identify and justify a class of models that flexibly capture a wide range of multilevel data. Motivated by the versatility of the mixture of experts (MoE) models in fitting regression data, in this article we extend upon the MoE and study a class of mixed MoE (MMoE) models for multilevel data. Under some regularity conditions, we prove that the MMoE is dense in the space of any continuous mixed effects models in the sense of weak convergence. As a result, the MMoE has a potential to accurately resemble almost all characteristics inherited in multilevel data, including the marginal distributions, dependence structures, regression links, random intercepts and random slopes. In a particular case where the multilevel data is hierarchical, we further show that a nested version of the MMoE universally approximates a broad range of dependence structures of the random effects among different factor levels.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 4 of its 49 references that are in the map. First posted on arXiv 30 Sep 2022.

Builds on

4 of 49 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning7 Jun

Other references · 3

  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  2. 2016A Universal Approximation Theorem for Mixture of Experts Modelscited by 17
  3. 2016Robust mixture of experts modeling using the t distributioncited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learningcited by 4
  2. 2026Mixture-of-Experts in Remote Sensing: A Surveycited by 0
  3. 2026Structure Aware Neural Architecture Search for Mixture of Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2017An Introduction to the Practical and Theoretical Aspects of Mixture-of-Experts Modeling
  2. 2018Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models
  3. 2019Regularized Estimation and Feature Selection in Mixtures of Generalized Linear Experts
  4. 2022Functional Mixtures-of-Experts
  5. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications
  6. 2026How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves
  7. 2024Learning Mixtures of Experts with EM: A Mirror Descent Perspective
  8. 2025Rethinking Sparse Mixture of Experts from a Unified Perspective