# Bias-variance decompositions: the exclusive privilege of Bregman divergences

arXiv: https://arxiv.org/abs/2501.18581 · HTML: https://mlsanity.com/paper/2501.18581/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 36 references that are in the map.
- First posted on arXiv 30 Jan 2025.

## Builds on (most cited first, 13 of 13)

- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Learning with Fenchel-Young Losses](https://mlsanity.com/paper/1901.02324/) (2019)
- [A Unified Theory of Diversity in Ensemble Learning](https://mlsanity.com/paper/2301.03962/) (2023)
- [A Modern Take on the Bias-Variance Tradeoff in Neural Networks](https://mlsanity.com/paper/1810.08591/) (2018)
- [Robust Bi-Tempered Logistic Loss Based on Bregman Divergences](https://mlsanity.com/paper/1906.03361/) (2019)
- [Uncertainty Estimates of Predictions via a General Bias-Variance Decomposition](https://mlsanity.com/paper/2210.12256/) (2022)
- [Optimal Training of Mean Variance Estimation Neural Networks](https://mlsanity.com/paper/2302.08875/) (2023)
- [Bregman Deviations of Generic Exponential Families](https://mlsanity.com/paper/2201.07306/) (2022)
- [Deep Divergence Learning](https://mlsanity.com/paper/2005.02612/) (2020)
- [On geodesic triangles with right angles in a dually flat space](https://mlsanity.com/paper/1910.03935/) (2019)
- [Variational Representations of Annealing Paths: Bregman Information under Monotonic Embedding](https://mlsanity.com/paper/2209.07481/) (2022)
- [Bias-Variance Decompositions for Margin Losses](https://mlsanity.com/paper/2204.12155/) (2022)
- [Generalized Bregman and Jensen divergences which include some f-divergences](https://mlsanity.com/paper/1808.06148/) (2018)

## Built on by (most cited first, 6 of 6)

- [When Does Predictive Inverse Dynamics Outperform Behavior Cloning?](https://mlsanity.com/paper/2601.21718/) (2026)
- [An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentation](https://mlsanity.com/paper/2510.25128/) (2025)
- [Subjective Risk Decomposition: A New View for Uncertainty Quantification](https://mlsanity.com/paper/2607.15196/) (2026)
- [Curved representational Bregman divergences and their applications](https://mlsanity.com/paper/2504.05654/) (2025)
- [When three experiments are better than two: Avoiding intractable correlated aleatoric uncertainty by leveraging a novel bias--variance tradeoff](https://mlsanity.com/paper/2509.04363/) (2025)
- [Supervised Distributional Reduction via Optimal Transport and Dependence Maximization](https://mlsanity.com/paper/2605.27619/) (2026)

## Abstract (arXiv)

Bias-variance decompositions are widely used to understand the generalization performance of machine learning models. While the squared error loss permits a straightforward decomposition, other loss functions - such as zero-one loss or L1 loss - either fail to sum bias and variance to the expected loss or rely on definitions that lack the essential properties of meaningful bias and variance. Recent research has shown that clean decompositions can be achieved for the broader class of Bregman divergences, with the cross-entropy loss as a special case. However, the necessary and sufficient conditions for these decompositions remain an open question. In this paper, we address this question by studying continuous, nonnegative loss functions that satisfy the identity of indiscernibles (zero loss if and only if the two arguments are identical), under mild regularity conditions. We prove that so-called g-Bregman or rho-tau divergences are the only such loss functions that have a clean bias-variance decomposition. A g-Bregman divergence can be transformed into a standard Bregman divergence through an invertible change of variables. This makes the squared Mahalanobis distance, up to such a variable transformation, the only symmetric loss function with a clean bias-variance decomposition. Consequently, common metrics such as zero-one and L1 losses cannot admit a clean bias-variance decomposition, explaining why previous attempts have failed. We also examine the impact of relaxing the restrictions on the loss functions and how this affects our results.