# Ensembles of Locally Independent Prediction Models

arXiv: https://arxiv.org/abs/1911.01291 · HTML: https://mlsanity.com/paper/1911.01291/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Nov 2019.

## Built on by (most cited first, 22 of 22)

- [Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalization](https://mlsanity.com/paper/2105.05612/) (2021)
- [Agree to Disagree: Diversity through Disagreement for Better Transferability](https://mlsanity.com/paper/2202.04414/) (2022)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [GLocalX -- From Local to Global Explanations of Black Box AI Models](https://mlsanity.com/paper/2101.07685/) (2021)
- [ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets](https://mlsanity.com/paper/2209.00613/) (2022)
- [A Path to Simpler Models Starts With Noise](https://mlsanity.com/paper/2310.19726/) (2023)
- [Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning](https://mlsanity.com/paper/2207.02598/) (2022)
- [Pathologies of Predictive Diversity in Deep Ensembles](https://mlsanity.com/paper/2302.00704/) (2023)
- [Controlling Directions Orthogonal to a Classifier](https://mlsanity.com/paper/2201.11259/) (2022)
- [Lyapunov Exponents for Diversity in Differentiable Games](https://mlsanity.com/paper/2112.14570/) (2021)
- [DIVE: Subgraph Disagreement for Graph Out-of-Distribution Generalization](https://mlsanity.com/paper/2408.04400/) (2024)
- [Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification](https://mlsanity.com/paper/2304.04042/) (2023)
- [On the Role of Randomization in Adversarially Robust Classification](https://mlsanity.com/paper/2302.07221/) (2023)
- [Learning Diverse Features in Vision Transformers for Improved Generalization](https://mlsanity.com/paper/2308.16274/) (2023)
- [Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks](https://mlsanity.com/paper/2310.02230/) (2023)
- [Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles](https://mlsanity.com/paper/2311.16176/) (2023)
- [Greedy Bayesian Posterior Approximation with Deep Ensembles](https://mlsanity.com/paper/2105.14275/) (2021)
- [Scalable Ensemble Diversification for OOD Generalization and Detection](https://mlsanity.com/paper/2409.16797/) (2024)
- [Adversarial attacks for mixtures of classifiers](https://mlsanity.com/paper/2307.10788/) (2023)
- [HGEN: Heterogeneous Graph Ensemble Networks](https://mlsanity.com/paper/2509.09843/) (2025)
- [UDUC: An Uncertainty-driven Approach for Learning-based Robust Control](https://mlsanity.com/paper/2405.02598/) (2024)
- [MLRS-PDS: A Meta-learning recommendation of dynamic ensemble selection pipelines](https://mlsanity.com/paper/2407.07528/) (2024)

## Abstract (arXiv)

Ensembles depend on diversity for improved performance. Many ensemble training methods, therefore, attempt to optimize for diversity, which they almost always define in terms of differences in training set predictions. In this paper, however, we demonstrate the diversity of predictions on the training set does not necessarily imply diversity under mild covariate shift, which can harm generalization in practical settings. To address this issue, we introduce a new diversity metric and associated method of training ensembles of models that extrapolate differently on local patches of the data manifold. Across a variety of synthetic and real-world tasks, we find that our method improves generalization and diversity in qualitatively novel ways, especially under data limits and covariate shift.