# Training independent subnetworks for robust prediction

arXiv: https://arxiv.org/abs/2010.06610 · HTML: https://mlsanity.com/paper/2010.06610/

## Key facts

- Cited by 177 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2020.

## Built on by (most cited first, 30 of 177)

- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models](https://mlsanity.com/paper/2305.12827/) (2023)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [Epistemic Neural Networks](https://mlsanity.com/paper/2107.08924/) (2021)
- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks](https://mlsanity.com/paper/2107.07455/) (2021)
- [Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognition](https://mlsanity.com/paper/2107.09249/) (2021)
- [Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning](https://mlsanity.com/paper/2106.04015/) (2021)
- [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://mlsanity.com/paper/2306.02050/) (2023)
- [Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification](https://mlsanity.com/paper/2110.14012/) (2021)
- [Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters](https://mlsanity.com/paper/2205.13703/) (2022)
- [Pre-trained Gaussian Processes for Bayesian Optimization](https://mlsanity.com/paper/2109.08215/) (2021)
- [Deep Ensembles Work, But Are They Necessary?](https://mlsanity.com/paper/2202.06985/) (2022)
- [On the Practicality of Deterministic Epistemic Uncertainty](https://mlsanity.com/paper/2107.00649/) (2021)
- [Packed-Ensembles for Efficient Uncertainty Estimation](https://mlsanity.com/paper/2210.09184/) (2022)
- [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022)
- [Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity](https://mlsanity.com/paper/2106.14568/) (2021)
- [Rethinking Confidence Calibration for Failure Prediction](https://mlsanity.com/paper/2303.02970/) (2023)
- [Neural Mean Discrepancy for Efficient Out-of-Distribution Detection](https://mlsanity.com/paper/2104.11408/) (2021)
- [PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness](https://mlsanity.com/paper/2312.02158/) (2023)
- [UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixup](https://mlsanity.com/paper/2209.08928/) (2022)
- [OmniPred: Language Models as Universal Regressors](https://mlsanity.com/paper/2402.14547/) (2024)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task models](https://mlsanity.com/paper/2210.09759/) (2022)
- [Diversity Matters When Learning From Ensembles](https://mlsanity.com/paper/2110.14149/) (2021)
- [RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustness](https://mlsanity.com/paper/2206.14502/) (2022)
- [FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classification](https://mlsanity.com/paper/2206.08671/) (2022)
- [Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning](https://mlsanity.com/paper/2111.03577/) (2021)
- [FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation](https://mlsanity.com/paper/2206.00050/) (2022)

## Abstract (arXiv)

Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over the original network. However, these methods still require multiple forward passes for prediction, leading to a significant computational cost. In this work, we show a surprising result: the benefits of using multiple predictions can be achieved `for free' under a single model's forward pass. In particular, we show that, using a multi-input multi-output (MIMO) configuration, one can utilize a single model's capacity to train multiple subnetworks that independently learn the task at hand. By ensembling the predictions made by the subnetworks, we improve model robustness without increasing compute. We observe a significant improvement in negative log-likelihood, accuracy, and calibration error on CIFAR10, CIFAR100, ImageNet, and their out-of-distribution variants compared to previous methods.