# Regularized Linear Regression for Binary Classification

arXiv: https://arxiv.org/abs/2311.02270 · HTML: https://mlsanity.com/paper/2311.02270/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 32 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Builds on (most cited first, 14 of 14)

- [A modern maximum-likelihood theory for high-dimensional logistic regression](https://mlsanity.com/paper/1803.06964/) (2018)
- [A Model of Double Descent for High-dimensional Binary Linear Classification](https://mlsanity.com/paper/1911.05822/) (2019)
- [Finite-sample Analysis of Interpolating Linear Classifiers in the Overparameterized Regime](https://mlsanity.com/paper/2004.12019/) (2020)
- [The role of regularization in classification of high-dimensional noisy Gaussian mixture](https://mlsanity.com/paper/2002.11544/) (2020)
- [Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions](https://mlsanity.com/paper/2106.03791/) (2021)
- [Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization](https://mlsanity.com/paper/2006.06560/) (2020)
- [Precise Statistical Analysis of Classification Accuracies for Adversarial Training](https://mlsanity.com/paper/2010.11213/) (2020)
- [The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression](https://mlsanity.com/paper/1804.09753/) (2018)
- [A framework to characterize performance of LASSO algorithms](https://mlsanity.com/paper/1303.7291/) (2013)
- [The Impact of Regularization on High-dimensional Logistic Regression](https://mlsanity.com/paper/1906.03761/) (2019)
- [Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View](https://mlsanity.com/paper/2011.07729/) (2020)
- [Sharp Asymptotics and Optimal Performance for Inference in Binary Models](https://mlsanity.com/paper/2002.07284/) (2020)
- [Fundamental Limits of Ridge-Regularized Empirical Risk Minimization in High Dimensions](https://mlsanity.com/paper/2006.08917/) (2020)
- [The Performance Analysis of Generalized Margin Maximizer (GMM) on Separable Data](https://mlsanity.com/paper/2010.15379/) (2020)

## Built on by (most cited first, 7 of 7)

- [Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data](https://mlsanity.com/paper/2502.04664/) (2025)
- [A Novel Gaussian Min-Max Theorem and its Applications](https://mlsanity.com/paper/2402.07356/) (2024)
- [The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model](https://mlsanity.com/paper/2501.16226/) (2025)
- [One-Bit Quantization and Sparsification for Multiclass Linear Classification with Strong Regularization](https://mlsanity.com/paper/2402.10474/) (2024)
- [One-Bit Quantization for Random Features Models](https://mlsanity.com/paper/2510.16250/) (2025)
- [Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime](https://mlsanity.com/paper/2603.10485/) (2026)
- [Precise Performance of Linear Denoisers in the Proportional Regime](https://mlsanity.com/paper/2603.18483/) (2026)

## Abstract (arXiv)

Regularized linear regression is a promising approach for binary classification problems in which the training set has noisy labels since the regularization term can help to avoid interpolating the mislabeled data points. In this paper we provide a systematic study of the effects of the regularization strength on the performance of linear classifiers that are trained to solve binary classification problems by minimizing a regularized least-squares objective. We consider the over-parametrized regime and assume that the classes are generated from a Gaussian Mixture Model (GMM) where a fraction c<frac12 of the training data is mislabeled. Under these assumptions, we rigorously analyze the classification errors resulting from the application of ridge, ℓ1, and ℓ_∞ regression. In particular, we demonstrate that ridge regression invariably improves the classification error. We prove that ℓ1 regularization induces sparsity and observe that in many cases one can sparsify the solution by up to two orders of magnitude without any considerable loss of performance, even though the GMM has no underlying sparsity structure. For ℓ_∞ regularization we show that, for large enough regularization strength, the optimal weights concentrate around two values of opposite sign. We observe that in many cases the corresponding "compression" of each weight to a single bit leads to very little loss in performance. These latter observations can have significant practical ramifications.