# What causes the test error? Going beyond bias-variance via ANOVA

arXiv: https://arxiv.org/abs/2010.05170 · HTML: https://mlsanity.com/paper/2010.05170/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2020.

## Built on by (most cited first, 28 of 28)

- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks](https://mlsanity.com/paper/2310.07891/) (2023)
- [Precise Learning Curves and Higher-Order Scaling Limits for Dot Product Kernel Regression](https://mlsanity.com/paper/2205.14846/) (2022)
- [Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks](https://mlsanity.com/paper/2109.09304/) (2021)
- [Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension](https://mlsanity.com/paper/2201.13383/) (2022)
- [Harmless interpolation in regression and classification with structured features](https://mlsanity.com/paper/2111.05198/) (2021)
- [Demystifying Disagreement-on-the-Line in High Dimensions](https://mlsanity.com/paper/2301.13371/) (2023)
- [Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition](https://mlsanity.com/paper/2103.09947/) (2021)
- [Redundant representations help generalization in wide neural networks](https://mlsanity.com/paper/2106.03485/) (2021)
- [Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond](https://mlsanity.com/paper/2411.00247/) (2024)
- [Risk and cross validation in ridge regression with correlated samples](https://mlsanity.com/paper/2408.04607/) (2024)
- [Bias-variance decomposition of overparameterized regression with random linear features](https://mlsanity.com/paper/2203.05443/) (2022)
- [Overparameterized random feature regression with nearly orthogonal data](https://mlsanity.com/paper/2211.06077/) (2022)
- [Understanding Model Ensemble in Transferable Adversarial Attack](https://mlsanity.com/paper/2410.06851/) (2024)
- [On the Double Descent of Random Features Models Trained with SGD](https://mlsanity.com/paper/2110.06910/) (2021)
- [Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structures](https://mlsanity.com/paper/2212.06757/) (2022)
- [The Geometry of Over-parameterized Regression and Adversarial Perturbations](https://mlsanity.com/paper/2103.14108/) (2021)
- [Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression](https://mlsanity.com/paper/2402.13622/) (2024)
- [Quantifying lottery tickets under label noise: accuracy, calibration, and complexity](https://mlsanity.com/paper/2306.12190/) (2023)
- [Signal-Plus-Noise Decomposition of Nonlinear Spiked Random Matrix Models](https://mlsanity.com/paper/2405.18274/) (2024)
- [Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding](https://mlsanity.com/paper/2602.18977/) (2026)
- [It's an Alignment, Not a Trade-off: Revisiting Bias and Variance in Deep Models](https://mlsanity.com/paper/2310.09250/) (2023)
- [No Free Lunch From Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions](https://mlsanity.com/paper/2412.05418/) (2024)
- [Gibbs-Based Information Criteria and the Over-Parameterized Regime](https://mlsanity.com/paper/2306.05583/) (2023)
- [Neural Tangent Kernels Motivate Graph Neural Networks with Cross-Covariance Graphs](https://mlsanity.com/paper/2310.10791/) (2023)
- [Understanding the Role of Optimization in Double Descent](https://mlsanity.com/paper/2312.03951/) (2023)
- [A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularization](https://mlsanity.com/paper/2508.17412/) (2025)
- [Feature Learning in Linear-Width Two-Layer Networks: Two vs. One Step of Gradient Descent](https://mlsanity.com/paper/2605.17767/) (2026)

## Abstract (arXiv)

Modern machine learning methods are often overparametrized, allowing adaptation to the data at a fine level. This can seem puzzling; in the worst case, such models do not need to generalize. This puzzle inspired a great amount of work, arguing when overparametrization reduces test error, in a phenomenon called "double descent". Recent work aimed to understand in greater depth why overparametrization is helpful for generalization. This leads to discovering the unimodality of variance as a function of the level of parametrization, and to decomposing the variance into that arising from label noise, initialization, and randomness in the training data to understand the sources of the error. In this work we develop a deeper understanding of this area. Specifically, we propose using the analysis of variance (ANOVA) to decompose the variance in the test error in a symmetric way, for studying the generalization performance of certain two-layer linear and non-linear networks. The advantage of the analysis of variance is that it reveals the effects of initialization, label noise, and training data more clearly than prior approaches. Moreover, we also study the monotonicity and unimodality of the variance components. While prior work studied the unimodality of the overall variance, we study the properties of each term in variance decomposition. One key insight is that in typical settings, the interaction between training samples and initialization can dominate the variance; surprisingly being larger than their marginal effect. Also, we characterize "phase transitions" where the variance changes from unimodal to monotone. On a technical level, we leverage advanced deterministic equivalent techniques for Haar random matrices, that -- to our knowledge -- have not yet been used in the area. We also verify our results in numerical simulations and on empirical data examples.