It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Models
Not on the map. Why?
Abstract (arXiv)
Classical wisdom in machine learning holds that the generalization error can be decomposed into bias and variance, and these two terms exhibit a trade-off. However, in this paper, we show that for an ensemble of deep learning based classification models, bias and variance are aligned at a sample level, where squared bias is approximately equal to variance for correctly classified sample points. We present empirical evidence confirming this phenomenon in a variety of deep learning models and datasets. Moreover, we study this phenomenon from two theoretical perspectives: calibration and neural collapse. We first show theoretically that under the assumption that the models are well calibrated, we can observe the bias-variance alignment. Second, starting from the picture provided by the neural collapse theory, we show an approximate correlation between bias and variance.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 13 Oct 2023.
Builds on
28 of 39 references.
References · 28
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
- 2019Measuring Calibration in Deep Learningcited by 443
- 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
- 2021Accuracy on the Line: On the Strong Correlation Between Out-of-Distribution and In-Distribution Generalizationcited by 248
- 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
- 2021Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Trainingcited by 166
- 2021On Calibration and Out-of-domain Generalizationcited by 134
16 more
- 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 116
- 2021Assessing Generalization of SGD via Disagreementcited by 116
- 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
- 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
- 2022Extended Unconstrained Features Model for Exploring Deep Neural Collapsecited by 101
- 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
- 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
- 2020Understanding Double Descent Requires a Fine-Grained Bias-Variance Decompositioncited by 75
- 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
- 2020Explicit regularization and implicit bias in deep network classifiers trained with the square losscited by 37
- 2020What causes the test error? Going beyond bias-variance via ANOVAcited by 28
- 2020Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized modelscited by 26
- 2022The Calibration Generalization Gapcited by 18
- 2022A Note on "Assessing Generalization of SGD via Disagreement"cited by 16
- 2023Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severitycited by 13
- 2022Bias-variance decomposition of overparameterized regression with random linear featurescited by 7
Built on by
Citing papers per year: 2024: 1.
1 paper.