Towards a Universal Gating Network for Mixtures of Experts
On the map: Mixture-of-experts → MoE theory and gating functions
Abstract (arXiv)
The combination and aggregation of knowledge from multiple neural networks can be commonly seen in the form of mixtures of experts. However, such combinations are usually done using networks trained on the same tasks, with little mention of the combination of heterogeneous pre-trained networks, especially in the data-free regime. This paper proposes multiple data-free methods for the combination of heterogeneous neural networks, ranging from the utilization of simple output logit statistics, to training specialized gating networks. The gating networks decide whether specific inputs belong to specific networks based on the nature of the expert activations generated. The experiments revealed that the gating networks, including the universal gating approach, constituted the most accurate approach, and therefore represent a pragmatic step towards applications with heterogeneous mixtures of experts in a data-free regime. The code for this project is hosted on github at https://github.com/cwkang1998/network-merging.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 8 of its 28 references that are in the map. First posted on arXiv 3 Nov 2020.
Builds on
8 of 28 references.
Closest ancestors · 1
Same direction, newest first
- 2018Granger-causal Attentive Mixtures of Experts: Learning Important Features with Neural Networks6 Feb
Other references · 7
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2018Multi-Source Domain Adaptation with Mixture of Expertscited by 62
- 2017The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classificationcited by 50
- 2018Hydra: an Ensemble of Convolutional Neural Networks for Geospatial Land Classificationcited by 12
- 2019ExpertMatcher: Automating ML Model Selection for Clients using Hidden Representationscited by 3
- 2019Fast and Flexible Image Blind Denoising via Competition of Expertscited by 3
- 2019ExpertMatcher: Automating ML Model Selection for Users in Resource Constrained Countriescited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2023Improving Expert Specialization in Mixture of Experts
- 2024k-Winners-Take-All Ensemble Neural Network
- 2026Mixture-of-Experts under Finite-Rate Gating: Communication--Generalization Trade-offs
- 2026A Heterogeneous Mixture of Experts Framework for Interpretable Machine Learning
- 2025On Linear Mode Connectivity of Mixture-of-Experts Architectures
- 2021Mixture of ELM based experts with trainable gating network
- 2025Training Diverse Graph Experts for Ensembles: A Systematic Empirical Study
- 2019Learning in Gated Neural Networks