# The Impact of Regularization on High-dimensional Logistic Regression

arXiv: https://arxiv.org/abs/1906.03761 · HTML: https://mlsanity.com/paper/1906.03761/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jun 2019.

## Built on by (most cited first, 30 of 35)

- [Label-Imbalanced and Group-Sensitive Classification under Overparameterization](https://mlsanity.com/paper/2103.01550/) (2021)
- [A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning](https://mlsanity.com/paper/2109.02355/) (2021)
- [Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions](https://mlsanity.com/paper/2106.03791/) (2021)
- [Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation](https://mlsanity.com/paper/2106.10865/) (2021)
- [Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data](https://mlsanity.com/paper/2502.04664/) (2025)
- [Phase Transitions in Transfer Learning for High-Dimensional Perceptrons](https://mlsanity.com/paper/2101.01918/) (2021)
- [On the Inherent Regularization Effects of Noise Injection During Training](https://mlsanity.com/paper/2102.07379/) (2021)
- [A Blessing of Dimensionality in Membership Inference through Regularization](https://mlsanity.com/paper/2205.14055/) (2022)
- [Rethinking Early Stopping: Refine, Then Calibrate](https://mlsanity.com/paper/2501.19195/) (2025)
- [Interpolation can hurt robust generalization even when there is no noise](https://mlsanity.com/paper/2108.02883/) (2021)
- [On the sample complexity of parameter estimation in logistic regression with normal design](https://mlsanity.com/paper/2307.04191/) (2023)
- [On the Generalization for Transfer Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2207.05377/) (2022)
- [A Non-Asymptotic Moreau Envelope Theory for High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/2210.12082/) (2022)
- [Sharp global convergence guarantees for iterative nonconvex optimization: A Gaussian process perspective](https://mlsanity.com/paper/2109.09859/) (2021)
- [Generalization for multiclass classification with overparameterized linear models](https://mlsanity.com/paper/2206.01399/) (2022)
- [Regularized Linear Regression for Binary Classification](https://mlsanity.com/paper/2311.02270/) (2023)
- [Universality of max-margin classifiers](https://mlsanity.com/paper/2310.00176/) (2023)
- [Observable adjustments in single-index models for regularized M-estimators](https://mlsanity.com/paper/2204.06990/) (2022)
- [High-dimensional logistic regression with missing data: Imputation, regularization, and universality](https://mlsanity.com/paper/2410.01093/) (2024)
- [SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression](https://mlsanity.com/paper/2103.12725/) (2021)
- [Correntropy-Based Logistic Regression with Automatic Relevance Determination for Robust Sparse Brain Activity Decoding](https://mlsanity.com/paper/2207.09693/) (2022)
- [General Loss Functions Lead to (Approximate) Interpolation in High Dimensions](https://mlsanity.com/paper/2303.07475/) (2023)
- [A Novel Gaussian Min-Max Theorem and its Applications](https://mlsanity.com/paper/2402.07356/) (2024)
- [The Breakdown of Gaussian Universality in Classification of High-dimensional Linear Factor Mixtures](https://mlsanity.com/paper/2410.05609/) (2024)
- [Random Features Model with General Convex Regularization: A Fine Grained Analysis with Precise Asymptotic Learning Curves](https://mlsanity.com/paper/2204.02678/) (2022)
- [Multinomial Logistic Regression: Asymptotic Normality on Null Covariates in High-Dimensions](https://mlsanity.com/paper/2305.17825/) (2023)
- [A Distributionally Robust Optimisation Approach to Fair Credit Scoring](https://mlsanity.com/paper/2402.01811/) (2024)
- [One-Bit Quantization and Sparsification for Multiclass Linear Classification with Strong Regularization](https://mlsanity.com/paper/2402.10474/) (2024)
- [Dimension-free uniform concentration bound for logistic regression](https://mlsanity.com/paper/2405.18055/) (2024)
- [RAPTOR: Ridge-Adaptive Logistic Probes](https://mlsanity.com/paper/2602.00158/) (2026)

## Abstract (arXiv)

Logistic regression is commonly used for modeling dichotomous outcomes. In the classical setting, where the number of observations is much larger than the number of parameters, properties of the maximum likelihood estimator in logistic regression are well understood. Recently, Sur and Candes have studied logistic regression in the high-dimensional regime, where the number of observations and parameters are comparable, and show, among other things, that the maximum likelihood estimator is biased. In the high-dimensional regime the underlying parameter vector is often structured (sparse, block-sparse, finite-alphabet, etc.) and so in this paper we study regularized logistic regression (RLR), where a convex regularizer that encourages the desired structure is added to the negative of the log-likelihood function. An advantage of RLR is that it allows parameter recovery even for instances where the (unconstrained) maximum likelihood estimate does not exist. We provide a precise analysis of the performance of RLR via the solution of a system of six nonlinear equations, through which any performance metric of interest (mean, mean-squared error, probability of support recovery, etc.) can be explicitly computed. Our results generalize those of Sur and Candes and we provide a detailed study for the cases of ℓ22-RLR and sparse (ℓ1-regularized) logistic regression. In both cases, we obtain explicit expressions for various performance metrics and can find the values of the regularizer parameter that optimizes the desired performance. The theory is validated by extensive numerical simulations across a range of parameter values and problem instances.