# On Power Laws in Deep Ensembles

arXiv: https://arxiv.org/abs/2007.08483 · HTML: https://mlsanity.com/paper/2007.08483/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jul 2020.

## Built on by (most cited first, 26 of 26)

- [MoViNets: Mobile Video Networks for Efficient Video Recognition](https://mlsanity.com/paper/2103.11511/) (2021)
- [Deep Ensembles Work, But Are They Necessary?](https://mlsanity.com/paper/2202.06985/) (2022)
- [Packed-Ensembles for Efficient Uncertainty Estimation](https://mlsanity.com/paper/2210.09184/) (2022)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworks](https://mlsanity.com/paper/2103.06132/) (2021)
- [Parallel Scaling Law for Language Models](https://mlsanity.com/paper/2505.10475/) (2025)
- [Pre-training under infinite compute](https://mlsanity.com/paper/2509.14786/) (2025)
- [Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning](https://mlsanity.com/paper/2111.03577/) (2021)
- [Pathologies of Predictive Diversity in Deep Ensembles](https://mlsanity.com/paper/2302.00704/) (2023)
- [Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models](https://mlsanity.com/paper/2303.08010/) (2023)
- [Accuracy-Privacy Trade-off in Deep Ensemble: A Membership Inference Perspective](https://mlsanity.com/paper/2105.05381/) (2021)
- [Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis](https://mlsanity.com/paper/2407.05385/) (2024)
- [Ensembling over Classifiers: a Bias-Variance Perspective](https://mlsanity.com/paper/2206.10566/) (2022)
- [Ex uno plures: Splitting One Model into an Ensemble of Subnetworks](https://mlsanity.com/paper/2106.04767/) (2021)
- [A Neural Scaling Law from Lottery Ticket Ensembling](https://mlsanity.com/paper/2310.02258/) (2023)
- [Position: Enough of Scaling LLMs! Lets Focus on Downscaling](https://mlsanity.com/paper/2505.00985/) (2025)
- [Is BatchEnsemble a Single Model? On Calibration and Diversity of Efficient Ensembles](https://mlsanity.com/paper/2601.16936/) (2026)
- [Population-size-Aware Policy Optimization for Mean-Field Games](https://mlsanity.com/paper/2302.03364/) (2023)
- [Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations](https://mlsanity.com/paper/2202.06499/) (2022)
- [Neural Diversity Regularizes Hallucinations in Language Models](https://mlsanity.com/paper/2510.20690/) (2025)
- [Effective training-time stacking for ensembling of deep neural networks](https://mlsanity.com/paper/2206.13491/) (2022)
- [Beyond Simple Averaging: Improving NLP Ensemble Performance with Topological-Data-Analysis-Based Weighting](https://mlsanity.com/paper/2402.14184/) (2024)
- [Linear Mode Connectivity under Data Shifts for Deep Ensembles of Image Classifiers](https://mlsanity.com/paper/2511.04514/) (2025)
- [Modeling Disagreement in Automatic Data Labelling for Semi-Supervised Learning in Clinical Natural Language Processing](https://mlsanity.com/paper/2205.14761/) (2022)
- [Non-Uniform Parameter-Wise Model Merging](https://mlsanity.com/paper/2412.15467/) (2024)
- [Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencoder](https://mlsanity.com/paper/2607.05025/) (2026)

## Abstract (arXiv)

Ensembles of deep neural networks are known to achieve state-of-the-art performance in uncertainty estimation and lead to accuracy improvement. In this work, we focus on a classification problem and investigate the behavior of both non-calibrated and calibrated negative log-likelihood (CNLL) of a deep ensemble as a function of the ensemble size and the member network size. We indicate the conditions under which CNLL follows a power law w.r.t. ensemble size or member network size, and analyze the dynamics of the parameters of the discovered power laws. Our important practical finding is that one large network may perform worse than an ensemble of several medium-size networks with the same total number of parameters (we call this ensemble a memory split). Using the detected power law-like dependencies, we can predict (1) the possible gain from the ensembling of networks with given structure, (2) the optimal memory split given a memory budget, based on a relatively small number of trained networks. We describe the memory split advantage effect in more details in arXiv:2005.07292