# Diversity inducing Information Bottleneck in Model Ensembles

arXiv: https://arxiv.org/abs/2003.04514 · HTML: https://mlsanity.com/paper/2003.04514/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2020.

## Built on by (most cited first, 27 of 27)

- [LookOut: Diverse Multi-Future Prediction and Planning for Self-Driving](https://mlsanity.com/paper/2101.06547/) (2021)
- [Graph Structure Learning with Variational Information Bottleneck](https://mlsanity.com/paper/2112.08903/) (2021)
- [A Unified Theory of Diversity in Ensemble Learning](https://mlsanity.com/paper/2301.03962/) (2023)
- [Agree to Disagree: Diversity through Disagreement for Better Transferability](https://mlsanity.com/paper/2202.04414/) (2022)
- [No One Representation to Rule Them All: Overlapping Features of Training Methods](https://mlsanity.com/paper/2110.12899/) (2021)
- [Diversify and Disambiguate: Learning From Underspecified Data](https://mlsanity.com/paper/2202.03418/) (2022)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model](https://mlsanity.com/paper/2310.17653/) (2023)
- [Orthogonal Ensemble Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/2105.10827/) (2021)
- [Pathologies of Predictive Diversity in Deep Ensembles](https://mlsanity.com/paper/2302.00704/) (2023)
- [Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEs](https://mlsanity.com/paper/2403.10642/) (2024)
- [Attribute-Modulated Generative Meta Learning for Zero-Shot Classification](https://mlsanity.com/paper/2104.10857/) (2021)
- [Lyapunov Exponents for Diversity in Differentiable Games](https://mlsanity.com/paper/2112.14570/) (2021)
- [Improving robustness and calibration in ensembles with diversity regularization](https://mlsanity.com/paper/2201.10908/) (2022)
- [A Too-Good-to-be-True Prior to Reduce Shortcut Reliance](https://mlsanity.com/paper/2102.06406/) (2021)
- [Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification](https://mlsanity.com/paper/2304.04042/) (2023)
- [Ex uno plures: Splitting One Model into an Ensemble of Subnetworks](https://mlsanity.com/paper/2106.04767/) (2021)
- [Greedy Bayesian Posterior Approximation with Deep Ensembles](https://mlsanity.com/paper/2105.14275/) (2021)
- [Variational Distillation for Multi-View Learning](https://mlsanity.com/paper/2206.09548/) (2022)
- [Causal Information Bottleneck Boosts Adversarial Robustness of Deep Neural Network](https://mlsanity.com/paper/2210.14229/) (2022)
- [Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learning](https://mlsanity.com/paper/2308.14705/) (2023)
- [Diversity-Aware Agnostic Ensemble of Sharpness Minimizers](https://mlsanity.com/paper/2403.13204/) (2024)
- [What Information Matters? Graph Out-of-Distribution Detection via Tri-Component Information Decomposition](https://mlsanity.com/paper/2605.13032/) (2026)
- [CGIBNet: Bandwidth-constrained Communication with Graph Information Bottleneck in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2112.10374/) (2021)
- [Quantifying Correlations of Machine Learning Models](https://mlsanity.com/paper/2502.03937/) (2025)
- [ACE and Diverse Generalization via Selective Disagreement](https://mlsanity.com/paper/2509.07955/) (2025)
- [Controllable Diversity in Normalization-Based Implicit Ensembles via Softmax-Temperature Modulation](https://mlsanity.com/paper/2607.23860/) (2026)

## Abstract (arXiv)

Although deep learning models have achieved state-of-the-art performance on a number of vision tasks, generalization over high dimensional multi-modal data, and reliable predictive uncertainty estimation are still active areas of research. Bayesian approaches including Bayesian Neural Nets (BNNs) do not scale well to modern computer vision tasks, as they are difficult to train, and have poor generalization under dataset-shift. This motivates the need for effective ensembles which can generalize and give reliable uncertainty estimates. In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning the stochastic latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. We evaluate our method on benchmark datasets: MNIST, CIFAR100, TinyImageNet and MIT Places 2, and compared to the most competitive baselines show significant improvements in classification accuracy, under a shift in the data distribution and in out-of-distribution detection. Code will be released in this url https://github.com/rvl-lab-utoronto/dibs