# Are Ensembles Getting Better all the Time?

arXiv: https://arxiv.org/abs/2311.17885 · HTML: https://mlsanity.com/paper/2311.17885/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 107 references that are in the map.
- First posted on arXiv 29 Nov 2023.

## Builds on (most cited first, 25 of 25)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Consistency of random forests](https://mlsanity.com/paper/1405.2881/) (2014)
- [The Shape of Learning Curves: a Review](https://mlsanity.com/paper/2103.10948/) (2021)
- [Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization](https://mlsanity.com/paper/1710.05179/) (2017)
- [Risk Bounds for the Majority Vote: From a PAC-Bayesian Analysis to a Learning Algorithm](https://mlsanity.com/paper/1503.08329/) (2015)
- [Packed-Ensembles for Efficient Uncertainty Estimation](https://mlsanity.com/paper/2210.09184/) (2022)
- [Bagging is an Optimal PAC Learner](https://mlsanity.com/paper/2212.02264/) (2022)
- [To tune or not to tune the number of trees in random forest?](https://mlsanity.com/paper/1705.05654/) (2017)
- [Diversity and Generalization in Neural Network Ensembles](https://mlsanity.com/paper/2110.13786/) (2021)
- [A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods](https://mlsanity.com/paper/2305.15027/) (2023)
- [Random-projection ensemble classification](https://mlsanity.com/paper/1504.04595/) (2015)
- [Variational Bayesian dropout: pitfalls and fixes](https://mlsanity.com/paper/1807.01969/) (2018)
- [When are ensembles really effective?](https://mlsanity.com/paper/2305.12313/) (2023)
- [Estimating the Algorithmic Variance of Randomized Ensembles via the Bootstrap](https://mlsanity.com/paper/1907.08742/) (2019)
- [Random projections: data perturbation for classification problems](https://mlsanity.com/paper/1911.10800/) (2019)
- [On Margins and Generalisation for Voting Classifiers](https://mlsanity.com/paper/2206.04607/) (2022)
- [Bounding Evidence and Estimating Log-Likelihood in VAE](https://mlsanity.com/paper/2206.09453/) (2022)
- [Towards efficient feature sharing in MIMO architectures](https://mlsanity.com/paper/2205.10139/) (2022)
- [Majority Voting and the Condorcet's Jury Theorem](https://mlsanity.com/paper/2002.03153/) (2020)
- [Uphill Roads to Variational Tightness: Monotonicity and Monte Carlo Objectives](https://mlsanity.com/paper/2201.10989/) (2022)

## Built on by (most cited first, 6 of 6)

- [On Learning-Curve Monotonicity for Maximum Likelihood Estimators](https://mlsanity.com/paper/2512.10220/) (2025)
- [OxEnsemble: Fair Ensembles for Low-Data Classification](https://mlsanity.com/paper/2512.09665/) (2025)
- [When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Models](https://mlsanity.com/paper/2601.11444/) (2026)
- [Beyond Mixtures and Products for Ensemble Aggregation: A Likelihood Perspective on Generalized Means](https://mlsanity.com/paper/2603.04204/) (2026)
- [Correcting Split Selection in Online Decision Trees via Anytime-Valid Inference](https://mlsanity.com/paper/2605.31239/) (2026)
- [A Distributional Optimisation Perspective on Combining Models in Deep Learning](https://mlsanity.com/paper/2609.24328/) (2026)

## Abstract (arXiv)

Ensemble methods combine the predictions of several base models. We study whether or not including more models always improves their average performance. This question depends on the kind of ensemble considered, as well as the predictive metric chosen. We focus on situations where all members of the ensemble are a priori expected to perform equally well, which is the case of several popular methods such as random forests or deep ensembles. In this setting, we show that ensembles are getting better all the time if, and only if, the considered loss function is convex. More precisely, in that case, the loss of the ensemble is a decreasing function of the number of models. When the loss function is nonconvex, we show a series of results that can be summarised as: ensembles of good models keep getting better, and ensembles of bad models keep getting worse. To this end, we prove a new result on the monotonicity of tail probabilities that may be of independent interest. We illustrate our results on a medical problem (diagnosing melanomas using neural nets) and a "wisdom of crowds" experiment (guessing the ratings of upcoming movies).