# An elementary analysis of ridge regression with random design

arXiv: https://arxiv.org/abs/2203.08564 · HTML: https://mlsanity.com/paper/2203.08564/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 43 references that are in the map.
- First posted on arXiv 16 Mar 2022.

## Builds on (most cited first, 8 of 8)

- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Benign overfitting in ridge regression](https://mlsanity.com/paper/2009.14286/) (2020)
- [Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)](https://mlsanity.com/paper/1306.2119/) (2013)
- [Optimal Rates For Regularization Of Statistical Inverse Learning Problems](https://mlsanity.com/paper/1604.04054/) (2016)
- [Exact minimax risk for linear least squares, and the lower tail of sample covariance matrices](https://mlsanity.com/paper/1912.10754/) (2019)
- [An improper estimator with optimal excess risk in misspecified density estimation and logistic regression](https://mlsanity.com/paper/1912.10784/) (2019)
- [Suboptimality of Constrained Least Squares and Improvements via Non-Linear Predictors](https://mlsanity.com/paper/2009.09304/) (2020)
- [Average Stability is Invariant to Data Preconditioning. Implications to Exp-concave Empirical Risk Minimization](https://mlsanity.com/paper/1601.04011/) (2016)

## Built on by (most cited first, 15 of 15)

- [High-dimensional analysis of double descent for linear regression with random projections](https://mlsanity.com/paper/2303.01372/) (2023)
- [The SSL Interplay: Augmentations, Inductive Bias, and Generalization](https://mlsanity.com/paper/2302.02774/) (2023)
- [Nonlinear Meta-Learning Can Guarantee Faster Rates](https://mlsanity.com/paper/2307.10870/) (2023)
- [Random features models: a way to study the success of naive imputation](https://mlsanity.com/paper/2402.03839/) (2024)
- [Fast kernel methods: Sobolev, physics-informed, and additive models](https://mlsanity.com/paper/2509.02649/) (2025)
- [Comparing regularisation paths of (conjugate) gradient estimators in ridge regression](https://mlsanity.com/paper/2503.05542/) (2025)
- [Kernel Density Machines](https://mlsanity.com/paper/2504.21419/) (2025)
- [How many samples are needed to leverage smoothness?](https://mlsanity.com/paper/2305.16014/) (2023)
- [Incentivized Exploration with Stochastic Covariates: A Two-Stage Mechanism Design for Recommender System](https://mlsanity.com/paper/2406.04374/) (2024)
- [Enhancing Federated Class-Incremental Learning via Spatial-Temporal Statistics Aggregation](https://mlsanity.com/paper/2506.01327/) (2025)
- [Generalisation and benign over-fitting for linear regression onto random functional covariates](https://mlsanity.com/paper/2508.13895/) (2025)
- [Handling Covariate Mismatch in Collaborative Linear Prediction](https://mlsanity.com/paper/2602.02083/) (2026)
- [Filter Learning for Subgraphs: Algebras and Performance Risk Bounds](https://mlsanity.com/paper/2607.21263/) (2026)
- [A Simple Approximation to the Distribution of the Ridge Regression Estimator](https://mlsanity.com/paper/2608.02539/) (2026)
- [Stochastic gradient descent with initial regularization](https://mlsanity.com/paper/2608.22953/) (2026)

## Abstract (arXiv)

In this note, we provide an elementary analysis of the prediction error of ridge regression with random design. The proof is short and self-contained. In particular, it bypasses the use of Rudelson's deviation inequality for covariance matrices, through a combination of exchangeability arguments, matrix perturbation and operator convexity.