# Lecture notes on ridge regression

arXiv: https://arxiv.org/abs/1509.09169 · HTML: https://mlsanity.com/paper/1509.09169/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2015.

## Built on by (most cited first, 16 of 16)

- [Towards a Better Understanding of Linear Models for Recommendation](https://mlsanity.com/paper/2105.12937/) (2021)
- [Optimal Training of Mean Variance Estimation Neural Networks](https://mlsanity.com/paper/2302.08875/) (2023)
- [Achieving Fairness with a Simple Ridge Penalty](https://mlsanity.com/paper/2105.13817/) (2021)
- [For high-dimensional hierarchical models, consider exchangeability of effects across covariates instead of across datasets](https://mlsanity.com/paper/2107.06428/) (2021)
- [OLR-WA Online Regression with Weighted Average](https://mlsanity.com/paper/2307.02804/) (2023)
- [Quantum Algorithms for the Pathwise Lasso](https://mlsanity.com/paper/2312.14141/) (2023)
- [A Geometric Unification of Distributionally Robust Covariance Estimators: Shrinking the Spectrum by Inflating the Ambiguity Set](https://mlsanity.com/paper/2405.20124/) (2024)
- [Distribution-dependent Generalization Bounds for Tuning Linear Regression Across Tasks](https://mlsanity.com/paper/2507.05084/) (2025)
- [Efficient Neural Network Training via Subset Pretraining](https://mlsanity.com/paper/2410.16523/) (2024)
- [Hamiltonian Monte Carlo Inference of Marginalized Linear Mixed-Effects Models](https://mlsanity.com/paper/2410.24079/) (2024)
- [Quantile deep learning models for multi-step ahead time series prediction](https://mlsanity.com/paper/2411.15674/) (2024)
- [Cluster Regularization via a Hierarchical Feature Regression](https://mlsanity.com/paper/2107.04831/) (2021)
- [Optimal Bias-Correction and Valid Inference in High-Dimensional Ridge Regression: A Closed-Form Solution](https://mlsanity.com/paper/2405.00424/) (2024)
- [Pathological Regularization Regimes in Classification Tasks](https://mlsanity.com/paper/2406.14731/) (2024)
- [Fundamentals of Regression](https://mlsanity.com/paper/2512.01920/) (2025)
- [Dissecting Jet-Tagger Through Mechanistic Interpretability](https://mlsanity.com/paper/2605.09881/) (2026)

## Abstract (arXiv)

The linear regression model cannot be fitted to high-dimensional data, as the high-dimensionality brings about empirical non-identifiability. Penalized regression overcomes this non-identifiability by augmentation of the loss function by a penalty (i.e. a function of regression coefficients). The ridge penalty is the sum of squared regression coefficients, giving rise to ridge regression. Here many aspect of ridge regression are reviewed e.g. moments, mean squared error, its equivalence to constrained estimation, and its relation to Bayesian regression. Finally, its behaviour and use are illustrated in simulation and on high-throughput data. Subsequently, ridge regression is generalized to allow for a more general penalty. The ridge penalization framework is then translated to logistic regression and its properties are shown to carry over. To contrast ridge penalized estimation, the final chapters introduce its lasso counterpart and generalizations thereof.