# Diversity and degrees of freedom in regression ensembles

arXiv: https://arxiv.org/abs/1803.00314 · HTML: https://mlsanity.com/paper/1803.00314/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2018.

## Built on by (most cited first, 6 of 6)

- [Ensembles of Randomized NNs for Pattern-based Time Series Forecasting](https://mlsanity.com/paper/2107.04091/) (2021)
- [Boosted Ensemble Learning based on Randomized NNs for Time Series Forecasting](https://mlsanity.com/paper/2203.00980/) (2022)
- [A hybrid ensemble method with negative correlation learning for regression](https://mlsanity.com/paper/2104.02317/) (2021)
- [Eliminating Multicollinearity Issues in Neural Network Ensembles: Incremental, Negatively Correlated, Optimal Convex Blending](https://mlsanity.com/paper/2104.14715/) (2021)
- [Combining Forecasts using Meta-Learning: A Comparative Study for Complex Seasonality](https://mlsanity.com/paper/2504.08940/) (2025)
- [Efficient Bayesian Deep Ensembles via Analytic Predictive Inference](https://mlsanity.com/paper/2607.06776/) (2026)

## Abstract (arXiv)

Ensemble methods are a cornerstone of modern machine learning. The performance of an ensemble depends crucially upon the level of diversity between its constituent learners. This paper establishes a connection between diversity and degrees of freedom (i.e. the capacity of the model), showing that diversity may be viewed as a form of inverse regularisation. This is achieved by focusing on a previously published algorithm Negative Correlation Learning (NCL), in which model diversity is explicitly encouraged through a diversity penalty term in the loss function. We provide an exact formula for the effective degrees of freedom in an NCL ensemble with fixed basis functions, showing that it is a continuous, convex and monotonically increasing function of the diversity parameter. We demonstrate a connection to Tikhonov regularisation and show that, with an appropriately chosen diversity parameter, an NCL ensemble can always outperform the unregularised ensemble in the presence of noise. We demonstrate the practical utility of our approach by deriving a method to efficiently tune the diversity parameter. Finally, we use a Monte-Carlo estimator to extend the connection between diversity and degrees of freedom to ensembles of deep neural networks.