Minimax Optimal Estimator and Improved Error Rate for the MLE in Logistic Regression with Gaussian Design

We study finite-sample parameter estimation in logistic regression with Gaussian design, where the goal is to estimate θ^*∈ ℝd with R=|θ^*|2≥ 1 from i.i.d. samples (xi,yi)i=1n, xi ~ N(0,Id), yimid xi ~ Bernoulli((1+exp(-xi^top θ^*))-1). In this paper, we provide the first minimax optimal estimator, and improve on the best known finite-sample error rate for the maximum likelihood estimator (MLE). These two accomplishments are due to a minimax optimal estimator for the parameter norm R. First, we establish the minimax lower bound Omega(√R3/n) for norm estimation. We then improve the best known norm estimation error rate of the MLE, i.e., O(√R3d/n) from Chardon, Lerasle and Mourtada (2024), to tildeO(√R3/n+R2d/n). The additional term, R2d/n, appears to be the intrinsic bias of the MLE, as evidenced by the high-dimensional asymptotic theory of Zhao, Sur and Candes (2022) and numerical examples. We show that, however, this additional term is not information-theoretically necessary. To this end, we construct an efficient debiased norm estimator that achieves the error rate O(√R3/n) and is therefore minimax optimal. Combining this with the optimal direction estimator given by the MLE, we establish the minimax optimal rate Theta(√Rd/n+√R3/n) for estimating θ^*, as well as the improved finite-sample error rate tildeO(√Rd/n+√R3/n+R2d/n) for the MLE. Numerical experiments demonstrate that the proposed minimax optimal estimators outperform the MLE.