# Generalization Error of Generalized Linear Models in High Dimensions

arXiv: https://arxiv.org/abs/2005.00180 · HTML: https://mlsanity.com/paper/2005.00180/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 May 2020.

## Built on by (most cited first, 25 of 25)

- [Label-Imbalanced and Group-Sensitive Classification under Overparameterization](https://mlsanity.com/paper/2103.01550/) (2021)
- [A phase transition between positional and semantic learning in a solvable model of dot-product attention](https://mlsanity.com/paper/2402.03902/) (2024)
- [Theoretical characterization of uncertainty in high-dimensional linear classification](https://mlsanity.com/paper/2202.03295/) (2022)
- [A Blessing of Dimensionality in Membership Inference through Regularization](https://mlsanity.com/paper/2205.14055/) (2022)
- [Implicit Bias of Linear RNNs](https://mlsanity.com/paper/2101.07833/) (2021)
- [Multiple Descent in the Multiple Random Feature Model](https://mlsanity.com/paper/2208.09897/) (2022)
- [BadVFL: Backdoor Attacks in Vertical Federated Learning](https://mlsanity.com/paper/2304.08847/) (2023)
- [Information bottleneck theory of high-dimensional regression: relevancy, efficiency and optimality](https://mlsanity.com/paper/2208.03848/) (2022)
- [Kernel Methods and Multi-layer Perceptrons Learn Linear Models in High Dimensions](https://mlsanity.com/paper/2201.08082/) (2022)
- [Generalization Error Bound for Quantum Machine Learning in NISQ Era -- A Survey](https://mlsanity.com/paper/2409.07626/) (2024)
- [Double-descent curves in neural networks: a new perspective using Gaussian processes](https://mlsanity.com/paper/2102.07238/) (2021)
- [Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression](https://mlsanity.com/paper/2402.13622/) (2024)
- [Asymptotics of Ridge Regression in Convolutional Models](https://mlsanity.com/paper/2103.04557/) (2021)
- [Towards Data-Algorithm Dependent Generalization: a Case Study on Overparameterized Linear Regression](https://mlsanity.com/paper/2202.06054/) (2022)
- [Sufficient-Statistic Memory AMP](https://mlsanity.com/paper/2112.15327/) (2021)
- [Monotonic Risk Relationships under Distribution Shifts for Regularized Risk Minimization](https://mlsanity.com/paper/2210.11589/) (2022)
- [Dimension-free uniform concentration bound for logistic regression](https://mlsanity.com/paper/2405.18055/) (2024)
- [Generalization vs. Specialization under Concept Shift](https://mlsanity.com/paper/2409.15582/) (2024)
- [Data-Dependent Generalization Bounds for Parameterized Quantum Models Under Noise](https://mlsanity.com/paper/2412.11451/) (2024)
- [Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensions](https://mlsanity.com/paper/2505.18046/) (2025)
- [Optimal Implicit Bias in Linear Regression](https://mlsanity.com/paper/2506.17187/) (2025)
- [Fundamental limits and algorithms for sparse linear regression with sublinear sparsity](https://mlsanity.com/paper/2101.11156/) (2021)
- [Accelerated Neural Network Training with Rooted Logistic Objectives](https://mlsanity.com/paper/2310.03890/) (2023)
- [Understanding the Role of Optimization in Double Descent](https://mlsanity.com/paper/2312.03951/) (2023)
- [Estimation of embedding vectors in high dimensions](https://mlsanity.com/paper/2312.07802/) (2023)

## Abstract (arXiv)

At the heart of machine learning lies the question of generalizability of learned rules over previously unseen data. While over-parameterized models based on neural networks are now ubiquitous in machine learning applications, our understanding of their generalization capabilities is incomplete. This task is made harder by the non-convexity of the underlying learning problems. We provide a general framework to characterize the asymptotic generalization error for single-layer neural networks (i.e., generalized linear models) with arbitrary non-linearities, making it applicable to regression as well as classification problems. This framework enables analyzing the effect of (i) over-parameterization and non-linearity during modeling; and (ii) choices of loss function, initialization, and regularizer during learning. Our model also captures mismatch between training and test distributions. As examples, we analyze a few special cases, namely linear regression and logistic regression. We are also able to rigorously and analytically explain the double descent phenomenon in generalized linear models.