# CoCoAFusE: Beyond Mixtures of Experts via Model Fusion

arXiv: https://arxiv.org/abs/2505.01105 · HTML: https://mlsanity.com/paper/2505.01105/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 15 of its 46 references that are in the map.
- First posted on arXiv 2 May 2025.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 15 of 15)

- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Pyro: Deep Universal Probabilistic Programming](https://mlsanity.com/paper/1810.09538/) (2018)
- [Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users](https://mlsanity.com/paper/2007.06823/) (2020)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning](https://mlsanity.com/paper/2106.03760/) (2021)
- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [Rank-normalization, folding, and localization: An improved widehatR for assessing convergence of MCMC](https://mlsanity.com/paper/1903.08008/) (2019)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Combining Modular Skills in Multitask Learning](https://mlsanity.com/paper/2202.13914/) (2022)
- [A Survey of Monte Carlo Methods for Parameter Estimation](https://mlsanity.com/paper/2107.11820/) (2021)
- [Revisiting Single-gated Mixtures of Experts](https://mlsanity.com/paper/2304.05497/) (2023)
- [Maximum Number of Modes of Gaussian Mixtures](https://mlsanity.com/paper/1702.05066/) (2017)
- [Bayesian dependent mixture models: A predictive comparison and survey](https://mlsanity.com/paper/2307.16298/) (2023)
- [A similarity-based Bayesian mixture-of-experts model](https://mlsanity.com/paper/2012.02130/) (2020)

## Built on by (most cited first, 1 of 1)

- [Explainable Condition Monitoring via Probabilistic Anomaly Detection Applied to Helicopter Transmissions](https://mlsanity.com/paper/2603.08130/) (2026)

## Abstract (arXiv)

Many learning problems involve multiple patterns and varying degrees of uncertainty dependent on the covariates. Advances in Deep Learning (DL) have addressed these issues by learning highly nonlinear input-output dependencies. However, model interpretability and Uncertainty Quantification (UQ) have often straggled behind. In this context, we introduce the Competitive/Collaborative Fusion of Experts (CoCoAFusE), a novel, Bayesian Covariates-Dependent Modeling technique. CoCoAFusE builds on the very philosophy behind Mixtures of Experts (MoEs), blending predictions from several simple sub-models (or "experts") to achieve high levels of expressiveness while retaining a substantial degree of local interpretability. Our formulation extends that of a classical Mixture of Experts by contemplating the fusion of the experts' distributions in addition to their more usual mixing (i.e., superimposition). Through this additional feature, CoCoAFusE better accommodates different scenarios for the intermediate behavior between generating mechanisms, resulting in tighter credible bounds on the response variable. Indeed, only resorting to mixing, as in classical MoEs, may lead to multimodality artifacts, especially over smooth transitions. Instead, CoCoAFusE can avoid these artifacts even under the same structure and priors for the experts, leading to greater expressiveness and flexibility in modeling. This new approach is showcased extensively on a suite of motivating numerical examples and a collection of real-data ones, demonstrating its efficacy in tackling complex regression problems where uncertainty is a key quantity of interest.