# Packed-Ensembles for Efficient Uncertainty Estimation

arXiv: https://arxiv.org/abs/2210.09184 · HTML: https://mlsanity.com/paper/2210.09184/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 79 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 30 of 57)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [A Simple Baseline for Bayesian Uncertainty in Deep Learning](https://mlsanity.com/paper/1902.02476/) (2019)
- [Bayesian Deep Learning and a Probabilistic Perspective of Generalization](https://mlsanity.com/paper/2002.08791/) (2020)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)

## Built on by (most cited first, 30 of 49)

- [TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensembling](https://mlsanity.com/paper/2410.24210/) (2024)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awareness](https://mlsanity.com/paper/2312.02158/) (2023)
- [Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?](https://mlsanity.com/paper/2308.06619/) (2023)
- [LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networks](https://mlsanity.com/paper/2405.14438/) (2024)
- [Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models](https://mlsanity.com/paper/2303.08010/) (2023)
- [Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models](https://mlsanity.com/paper/2312.15297/) (2023)
- [A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors](https://mlsanity.com/paper/2310.08287/) (2023)
- [Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It](https://mlsanity.com/paper/2403.14715/) (2024)
- [Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding](https://mlsanity.com/paper/2403.17010/) (2024)
- [Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation](https://mlsanity.com/paper/2412.03178/) (2024)
- [Learning Credal Ensembles via Distributionally Robust Optimization](https://mlsanity.com/paper/2602.08470/) (2026)
- [Are Ensembles Getting Better all the Time?](https://mlsanity.com/paper/2311.17885/) (2023)
- [Regularized Multi-Decoder Ensemble for an Error-Aware Scene Representation Network](https://mlsanity.com/paper/2407.19082/) (2024)
- [OCCUQ: Exploring Efficient Uncertainty Quantification for 3D Occupancy Prediction](https://mlsanity.com/paper/2503.10605/) (2025)
- [Uncertainty-Aware Reward Modeling for Stable RLHF](https://mlsanity.com/paper/2606.19818/) (2026)
- [The Robust Semantic Segmentation UNCV2023 Challenge Results](https://mlsanity.com/paper/2309.15478/) (2023)
- [Tailoring Mixup to Data for Calibration](https://mlsanity.com/paper/2311.01434/) (2023)
- [Uncertainty Estimation for 3D Object Detection via Evidential Learning](https://mlsanity.com/paper/2410.23910/) (2024)
- [Ensembling Pruned Attention Heads For Uncertainty-Aware Efficient Transformers](https://mlsanity.com/paper/2510.18358/) (2025)
- [Hierarchical Light Transformer Ensembles for Multimodal Trajectory Forecasting](https://mlsanity.com/paper/2403.17678/) (2024)
- [Information-driven Affordance Discovery for Efficient Robotic Manipulation](https://mlsanity.com/paper/2405.03865/) (2024)
- [Torch-Uncertainty: A Deep Learning Framework for Uncertainty Quantification](https://mlsanity.com/paper/2511.10282/) (2025)
- [Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles](https://mlsanity.com/paper/2601.16936/) (2026)
- [Audio-Driven Talking Face Video Generation with Joint Uncertainty Learning](https://mlsanity.com/paper/2504.18810/) (2025)
- [Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes](https://mlsanity.com/paper/2512.14177/) (2025)
- [Learning to Generate Training Datasets for Robust Semantic Segmentation](https://mlsanity.com/paper/2308.02535/) (2023)
- [How does Bayesian Sampling help Membership Inference Attacks?](https://mlsanity.com/paper/2503.07482/) (2025)
- [A Closer Look at Cross-Domain Few-Shot Object Detection: Fine-Tuning Matters and Parallel Decoder Helps](https://mlsanity.com/paper/2603.28182/) (2026)
- [Network Fission Ensembles for Low-Cost Self-Ensembles](https://mlsanity.com/paper/2408.02301/) (2024)

## Abstract (arXiv)

Deep Ensembles (DE) are a prominent approach for achieving excellent performance on key metrics such as accuracy, calibration, uncertainty estimation, and out-of-distribution detection. However, hardware limitations of real-world systems constrain to smaller ensembles and lower-capacity networks, significantly deteriorating their performance and properties. We introduce Packed-Ensembles (PE), a strategy to design and train lightweight structured ensembles by carefully modulating the dimension of their encoding space. We leverage grouped convolutions to parallelize the ensemble into a single shared backbone and forward pass to improve training and inference speeds. PE is designed to operate within the memory limits of a standard neural network. Our extensive research indicates that PE accurately preserves the properties of DE, such as diversity, and performs equally well in terms of accuracy, calibration, out-of-distribution detection, and robustness to distribution shift. We make our code available at https://github.com/ENSTA-U2IS/torch-uncertainty.