mlsanityData to 7 Oct 2026

Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction

arXiv 2604.21203 ↗First posted 23 Apr 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of n(α-1)/2 √log n, outperforming existing Hessian-free alternatives.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 49 references that are in the map. First posted on arXiv 23 Apr 2026.

Builds on

14 of 49 references.

References · 14

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  2. 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
  3. 2020Online Covariance Matrix Estimation in Stochastic Gradient Descentcited by 37
  4. 2021Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scalingcited by 28
  5. 2017Statistical inference using SGDcited by 21
  6. 2020Statistical Inference for Online Decision Making via Stochastic Gradient Descentcited by 17
  7. 2018First-order Newton-type Estimator for Distributed Estimation and Inferencecited by 16
  8. 2018HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximationcited by 13
  9. 2023Online Bootstrap Inference with Nonconvex Stochastic Gradient Descent Estimatorcited by 12
  10. 2022Statistical Inference of Constrained Stochastic Optimization via Sketched Sequential Quadratic Programmingcited by 10
  11. 2024High Confidence Level Inference is Almost Free using Parallel Stochastic Optimizationcited by 9
  12. 2025Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descentcited by 8
2 more
  1. 2022Covariance Estimators for the ROOT-SGD Algorithm in Online Learningcited by 5
  2. 2025Online Covariance Estimation in Nonsmooth Stochastic Approximationcited by 4

Built on by

Not cited yet.