# Finite-Sample Performance of Gradient Descent in Logistic Regression with Gaussian Design

arXiv: https://arxiv.org/abs/2606.21683 · HTML: https://mlsanity.com/paper/2606.21683/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 40 references that are in the map.
- First posted on arXiv 19 Jun 2026.

## Builds on (most cited first, 18 of 18)

- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Convergence of Gradient Descent on Separable Data](https://mlsanity.com/paper/1803.01905/) (2018)
- [Phase Retrieval using Alternating Minimization](https://mlsanity.com/paper/1306.0160/) (2013)
- [The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression](https://mlsanity.com/paper/1804.09753/) (2018)
- [Implicit Bias of Gradient Descent for Logistic Regression at the Edge of Stability](https://mlsanity.com/paper/2305.11788/) (2023)
- [Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency](https://mlsanity.com/paper/2402.15926/) (2024)
- [Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes](https://mlsanity.com/paper/2406.05033/) (2024)
- [On the sample complexity of parameter estimation in logistic regression with normal design](https://mlsanity.com/paper/2307.04191/) (2023)
- [Binary Iterative Hard Thresholding Converges with Optimal Number of Measurements for 1-Bit Compressed Sensing](https://mlsanity.com/paper/2207.03427/) (2022)
- [Large Stepsizes Accelerate Gradient Descent for Regularized Logistic Regression](https://mlsanity.com/paper/2506.02336/) (2025)
- [Model selection and minimax estimation in generalized linear models](https://mlsanity.com/paper/1409.8491/) (2014)
- [Gradient Descent Converges Linearly for Logistic Regression on Separable Data](https://mlsanity.com/paper/2306.14381/) (2023)
- [Finite-sample performance of the maximum likelihood estimator in logistic regression](https://mlsanity.com/paper/2411.02137/) (2024)
- [From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizes](https://mlsanity.com/paper/2412.08424/) (2024)
- [Gradient Descent on Logistic Regression: Do Large Step-Sizes Work with Data on the Sphere?](https://mlsanity.com/paper/2507.11228/) (2025)
- [Learning sparse generalized linear models with binary outcomes via iterative hard thresholding](https://mlsanity.com/paper/2502.18393/) (2025)
- [Low-Rank Tensor Estimation from Nonlinear Observations: A Unified Framework](https://mlsanity.com/paper/2510.16965/) (2025)

## Built on by (most cited first, 1 of 1)

- [Minimax Optimal Estimator and Improved Error Rate for the MLE in Logistic Regression with Gaussian Design](https://mlsanity.com/paper/2608.17260/) (2026)

## Abstract (arXiv)

We consider the parameter estimation problem in logistic regression with Gaussian design: the estimation of a fixed unknown parameter θ^*∈ ℝd (|θ^*|2≥ 1) from n i.i.d. samples (xi,yi)i=1n, where xi~ N(0,Id) and yi|xi ~ rm Bernoulli(1/(1+exp(-xi^top θ^*))). Our main aim is to characterize the finite-sample estimation performance and convergence behavior of gradient descent (GD) on the maximum likelihood objective (i.e., the logistic loss). Under small O(1) stepsize and 0 initialization, we show that GD linearly converges to a small neighborhood of θ^* achieving an ℓ2 error of order O(√|θ^*|25d/n). This substantially goes beyond existing theoretical results that lack non-asymptotic estimation error rate and exhibit much slower parameter convergence. We also establish a faster local linear convergence to the same statistical error under a large Theta(|θ^*|2) stepsize. The main technical component is to show that the gradient of the logistic loss satisfies a certain approximate invertibility condition (AIC). To that end, we uniformly control the deviation of the gradient from its population counterpart by covering and peeling arguments, and then show that the population GD is a contraction by a delicate analysis based on the eigenvalues of population Hessian matrices. Finally, we build upon the recent work Matsumoto and Mazumdar (2025) and devise a novel efficient estimator that attains a sharper rate in high dimensions. This indicates that the existing non-asymptotic guarantees exhibit sub-optimal dependence on |θ^*|2, and that in many regimes Theta(√|θ^*|2d/n) is the tight estimation error rate. Numerical examples are provided to corroborate our theoretical results.