mlsanityData to 7 Oct 2026

Training independent subnetworks for robust prediction

arXiv 2010.06610 ↗First posted 13 Oct 2020cs.LG cs.CV stat.MLcited by 177

Not on the map. Why?

Abstract (arXiv)

Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over the original network. However, these methods still require multiple forward passes for prediction, leading to a significant computational cost. In this work, we show a surprising result: the benefits of using multiple predictions can be achieved `for free' under a single model's forward pass. In particular, we show that, using a multi-input multi-output (MIMO) configuration, one can utilize a single model's capacity to train multiple subnetworks that independently learn the task at hand. By ensembling the predictions made by the subnetworks, we improve model robustness without increasing compute. We observe a significant improvement in negative log-likelihood, accuracy, and calibration error on CIFAR10, CIFAR100, ImageNet, and their out-of-distribution variants compared to previous methods.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 34; 2022: 45; 2023: 38; 2024: 24; 2025: 22; 2026: 14.

177 papers.

Sort:

By area:

  1. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  2. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  3. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  4. 2022Trusted Multi-View Classification with Dynamic Evidential Fusioncited by 167
  5. 2021Epistemic Neural Networkscited by 120
  6. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 105
  7. 2021Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Taskscited by 100
  8. 2021Self-Supervised Aggregation of Diverse Experts for Test-Agnostic Long-Tailed Recognitioncited by 95
  9. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  10. 2023Provable Dynamic Fusion for Low-Quality Multimodal Datacited by 81
  11. 2021Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classificationcited by 79
  12. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 74
38 more
  1. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  2. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  3. 2021On the Practicality of Deterministic Epistemic Uncertaintycited by 62
  4. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  5. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  6. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  7. 2023Rethinking Confidence Calibration for Failure Predictioncited by 43
  8. 2021Neural Mean Discrepancy for Efficient Out-of-Distribution Detectioncited by 42
  9. 2023PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awarenesscited by 42
  10. 2022UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixupcited by 34
  11. 2024OmniPred: Language Models as Universal Regressorscited by 34
  12. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  13. 2022Pareto Manifold Learning: Tackling multiple tasks via ensembles of single-task modelscited by 29
  14. 2021Diversity Matters When Learning From Ensemblescited by 27
  15. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  16. 2022FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classificationcited by 26
  17. 2021Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learningcited by 25
  18. 2022FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulationcited by 24
  19. 2022Shifts 2.0: Extending The Dataset of Real Distributional Shiftscited by 22
  20. 2024LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networkscited by 20
  21. 2021Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targetscited by 19
  22. 2021Robust Semantic Segmentation with Superpixel-Mixcited by 19
  23. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  24. 2022Weighted Ensemble Self-Supervised Learningcited by 19
  25. 2022DataMUX: Data Multiplexing for Neural Networkscited by 18
  26. 2022Continual Learning Beyond a Single Modelcited by 18
  27. 2023Calibrating Multimodal Learningcited by 17
  28. 2021NOMU: Neural Optimization-based Model Uncertaintycited by 16
  29. 2021Efficient Self-Ensemble for Semantic Segmentationcited by 16
  30. 2022Can we achieve robustness from data alone?cited by 16
  31. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  32. 2023Multiclass Confidence and Localization Calibration for Object Detectioncited by 16
  33. 2025Efficient Model Editing with Task-Localized Sparse Fine-tuningcited by 16
  34. 2021Deep Learning for Bayesian Optimization of Scientific Problems with High-Dimensional Structurecited by 15
  35. 2023Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Modelscited by 15
  36. 2023Uncertainty-based Fairness Measurescited by 15
  37. 2023Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Modelscited by 15
  38. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14

50 most cited of 177 All 177 (JSON)