# Towards a Universal Gating Network for Mixtures of Experts

arXiv: https://arxiv.org/abs/2011.01613 · HTML: https://mlsanity.com/paper/2011.01613/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 8 of its 28 references that are in the map.
- First posted on arXiv 3 Nov 2020.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 8 of 8)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Multi-Source Domain Adaptation with Mixture of Experts](https://mlsanity.com/paper/1809.02256/) (2018)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [Granger-causal Attentive Mixtures of Experts: Learning Important Features with Neural Networks](https://mlsanity.com/paper/1802.02195/) (2018)
- [Hydra: an Ensemble of Convolutional Neural Networks for Geospatial Land Classification](https://mlsanity.com/paper/1802.03518/) (2018)
- [ExpertMatcher: Automating ML Model Selection for Clients using Hidden Representations](https://mlsanity.com/paper/1910.03731/) (2019)
- [Fast and Flexible Image Blind Denoising via Competition of Experts](https://mlsanity.com/paper/1911.08724/) (2019)
- [ExpertMatcher: Automating ML Model Selection for Users in Resource Constrained Countries](https://mlsanity.com/paper/1910.02312/) (2019)

## Abstract (arXiv)

The combination and aggregation of knowledge from multiple neural networks can be commonly seen in the form of mixtures of experts. However, such combinations are usually done using networks trained on the same tasks, with little mention of the combination of heterogeneous pre-trained networks, especially in the data-free regime. This paper proposes multiple data-free methods for the combination of heterogeneous neural networks, ranging from the utilization of simple output logit statistics, to training specialized gating networks. The gating networks decide whether specific inputs belong to specific networks based on the nature of the expert activations generated. The experiments revealed that the gating networks, including the universal gating approach, constituted the most accurate approach, and therefore represent a pragmatic step towards applications with heterogeneous mixtures of experts in a data-free regime. The code for this project is hosted on github at https://github.com/cwkang1998/network-merging.