# Gradient-free variational learning with conditional mixture networks

arXiv: https://arxiv.org/abs/2408.16429 · HTML: https://mlsanity.com/paper/2408.16429/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 29 Aug 2024.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 21 of 21)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Laplace Redux -- Effortless Bayesian Deep Learning](https://mlsanity.com/paper/2106.14806/) (2021)
- [Black Box Variational Inference](https://mlsanity.com/paper/1401.0118/) (2013)
- [What Are Bayesian Neural Network Posteriors Really Like?](https://mlsanity.com/paper/2104.14421/) (2021)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [Learning Factored Representations in a Deep Mixture of Experts](https://mlsanity.com/paper/1312.4314/) (2013)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC](https://mlsanity.com/paper/1507.04544/) (2015)
- [Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting](https://mlsanity.com/paper/2010.06772/) (2020)
- [Understanding predictive information criteria for Bayesian models](https://mlsanity.com/paper/1307.5928/) (2013)
- [Bayesian Workflow](https://mlsanity.com/paper/2011.01808/) (2020)
- [Deep Gaussian Mixture Models](https://mlsanity.com/paper/1711.06929/) (2017)
- [Dependent Multinomial Models Made Easy: Stick Breaking with the P'olya-Gamma Augmentation](https://mlsanity.com/paper/1506.05843/) (2015)
- [A Universal Approximation Theorem for Mixture of Experts Models](https://mlsanity.com/paper/1602.03683/) (2016)
- [Conditionally conjugate mean-field variational Bayes for logistic models](https://mlsanity.com/paper/1711.06999/) (2017)
- [Calibrating Deep Neural Network Classifiers on Out-of-Distribution Datasets](https://mlsanity.com/paper/2006.08914/) (2020)
- [Ultimate P'olya Gamma Samplers -- Efficient MCMC for possibly imbalanced binary and categorical data](https://mlsanity.com/paper/2011.06898/) (2020)
- [Data Augementation with Polya Inverse Gamma](https://mlsanity.com/paper/1905.12141/) (2019)
- [Probabilistic Transformers](https://mlsanity.com/paper/2010.15583/) (2020)

## Built on by (most cited first, 4 of 4)

- [Navigation under uncertainty: Trajectory prediction and occlusion reasoning with switching dynamical systems](https://mlsanity.com/paper/2410.10653/) (2024)
- [AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models](https://mlsanity.com/paper/2505.24784/) (2025)
- [Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts](https://mlsanity.com/paper/2601.03577/) (2026)
- [Bayesian Predictive Coding](https://mlsanity.com/paper/2503.24016/) (2025)

## Abstract (arXiv)

Balancing computational efficiency with robust predictive performance is crucial in supervised learning, especially for critical applications. Standard deep learning models, while accurate and scalable, often lack probabilistic features like calibrated predictions and uncertainty quantification. Bayesian methods address these issues but can be computationally expensive as model and data complexity increase. Previous work shows that fast variational methods can reduce the compute requirements of Bayesian methods by eliminating the need for gradient computation or sampling, but are often limited to simple models. We introduce CAVI-CMN, a fast, gradient-free variational method for training conditional mixture networks (CMNs), a probabilistic variant of the mixture-of-experts (MoE) model. CMNs are composed of linear experts and a softmax gating network. By exploiting conditional conjugacy and P'olya-Gamma augmentation, we furnish Gaussian likelihoods for the weights of both the linear layers and the gating network. This enables efficient variational updates using coordinate ascent variational inference (CAVI), avoiding traditional gradient-based optimization. We validate this approach by training two-layer CMNs on standard classification benchmarks from the UCI repository. CAVI-CMN achieves competitive and often superior predictive accuracy compared to maximum likelihood estimation (MLE) with backpropagation, while maintaining competitive runtime and full posterior distributions over all model parameters. Moreover, as input size or the number of experts increases, computation time scales competitively with MLE and other gradient-based solutions like black-box variational inference (BBVI), making CAVI-CMN a promising tool for deep, fast, and gradient-free Bayesian networks.