# Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning

arXiv: https://arxiv.org/abs/2407.02721 · HTML: https://mlsanity.com/paper/2407.02721/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm](https://mlsanity.com/paper/1608.04471/) (2016)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [Stochastic Gradient Hamiltonian Monte Carlo](https://mlsanity.com/paper/1402.4102/) (2014)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam](https://mlsanity.com/paper/1806.04854/) (2018)
- [A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks](https://mlsanity.com/paper/1912.10481/) (2019)
- [Feature-map-level Online Adversarial Knowledge Distillation](https://mlsanity.com/paper/2002.01775/) (2020)
- [Bayesian Dark Knowledge](https://mlsanity.com/paper/1506.04416/) (2015)
- [Feature Fusion for Online Mutual Knowledge Distillation](https://mlsanity.com/paper/1904.09058/) (2019)
- [Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning](https://mlsanity.com/paper/1907.00865/) (2019)
- [Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors](https://mlsanity.com/paper/1806.05975/) (2018)
- [Adversarial Distillation of Bayesian Neural Network Posteriors](https://mlsanity.com/paper/1806.10317/) (2018)
- [Gaussian variational approximation with a factor covariance structure](https://mlsanity.com/paper/1701.03208/) (2017)
- [Walsh-Hadamard Variational Inference for Bayesian Deep Learning](https://mlsanity.com/paper/1905.11248/) (2019)

## Built on by (most cited first, 2 of 2)

- [Stochastic Weight Sharing for Bayesian Neural Networks](https://mlsanity.com/paper/2505.17856/) (2025)
- [Enhancing Graph Neural Networks: A Mutual Learning Approach](https://mlsanity.com/paper/2510.19223/) (2025)

## Abstract (arXiv)

Bayesian Neural Networks (BNNs) offer probability distributions for model parameters, enabling uncertainty quantification in predictions. However, they often underperform compared to deterministic neural networks. Utilizing mutual learning can effectively enhance the performance of peer BNNs. In this paper, we propose a novel approach to improve BNNs performance through deep mutual learning. The proposed approaches aim to increase diversity in both network parameter distributions and feature distributions, promoting peer networks to acquire distinct features that capture different characteristics of the input, which enhances the effectiveness of mutual learning. Experimental results demonstrate significant improvements in the classification accuracy, negative log-likelihood, and expected calibration error when compared to traditional mutual learning for BNNs.