# Ensemble Performance Through the Lens of Linear Independence of Classifier Votes in Data Streams

arXiv: https://arxiv.org/abs/2511.21465 · HTML: https://mlsanity.com/paper/2511.21465/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 53 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Builds on (most cited first, 7 of 7)

- [Scikit-Multiflow: A Multi-output Streaming Framework](https://mlsanity.com/paper/1807.04662/) (2018)
- [To tune or not to tune the number of trees in random forest?](https://mlsanity.com/paper/1705.05654/) (2017)
- [Less Is More: A Comprehensive Framework for the Number of Components of Ensemble Classifiers](https://mlsanity.com/paper/1709.02925/) (2017)
- [On-the-Fly Ensemble Pruning in Evolving Data Streams](https://mlsanity.com/paper/2109.07611/) (2021)
- [DynED: Dynamic Ensemble Diversification in Data Stream Classification](https://mlsanity.com/paper/2308.10807/) (2023)
- [Selecting a number of voters for a voting ensemble](https://mlsanity.com/paper/2104.11833/) (2021)
- [A Dataset-Level Geometric Framework for Ensemble Classifiers](https://mlsanity.com/paper/2106.08658/) (2021)

## Abstract (arXiv)

Ensemble learning improves classification performance by combining multiple base classifiers. While increasing the number of classifiers generally enhances accuracy, excessively large ensembles can lead to computational inefficiency and diminishing returns. This paper investigates the relationship between ensemble size and performance through the lens of linear independence among classifier votes in data streams. We propose that ensembles composed of linearly independent classifiers maximize representational capacity, particularly under a geometric model. We then generalize the importance of linear independence to the weighted majority voting problem. By modeling the probability of achieving linear independence among classifier outputs, we derive a theoretical framework that explains the trade-off between ensemble size and accuracy. Our analysis leads to a theoretical estimate of the ensemble size required to achieve a user-specified probability of linear independence. We validate our theory through experiments on both real-world and synthetic datasets using two ensemble methods, OzaBagging and GOOWE. Our results confirm that this theoretical estimate effectively identifies the point of performance saturation for robust ensembles like OzaBagging. Conversely, for complex weighting schemes like GOOWE, our framework reveals that high theoretical diversity can trigger algorithmic instability. Our implementation is publicly available to support reproducibility and future research.