mlsanityData to 14 Oct 2026

Statistical Inference for Model Parameters in Stochastic Gradient Descent

arXiv 1610.08637 ↗First posted 27 Oct 2016stat.MLcited by 71

Not on the map. Why?

Abstract (arXiv)

The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistical inference of true model parameters based on SGD when the population loss function is strongly convex and satisfies certain smoothness conditions. Our main contributions are two-fold. First, in the fixed dimension setup, we propose two consistent estimators of the asymptotic covariance of the average iterate from SGD: (1) a plug-in estimator, and (2) a batch-means estimator, which is computationally more efficient and only uses the iterates from SGD. Both proposed estimators allow us to construct asymptotically exact confidence intervals and hypothesis tests. Second, for high-dimensional linear regression, using a variant of the SGD algorithm, we construct a debiased estimator of each regression coefficient that is asymptotically normal. This gives a one-pass algorithm for computing both the sparse regression coefficients and confidence intervals, which is computationally attractive and applicable to online data.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Oct 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 13; 2022: 8; 2023: 15; 2024: 11; 2025: 12; 2026: 12.

71 papers.

Sort:

By area:

  1. 2021Convergence Rates of Stochastic Gradient Descent under Infinite Noise Variancecited by 35
  2. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  3. 2021Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scalingcited by 28
  4. 2021Tight High Probability Bounds for Linear Stochastic Approximation with Fixed Stepsizecited by 27
  5. 2021Online Bootstrap Inference For Policy Evaluation in Reinforcement Learningcited by 26
  6. 2023Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Datacited by 20
  7. 2024Gaussian Approximation and Multiplier Bootstrap for Polyak-Ruppert Averaged Linear Stochastic Approximation with Applications to TD Learningcited by 19
  8. 2022Optimal variance-reduced stochastic approximation in Banach spacescited by 17
  9. 2022Strong Transferable Adversarial Attacks via Ensembled Asymptotically Normal Distribution Learningcited by 15
  10. 2021Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methodscited by 14
  11. 2024Private Gradient Descent for Linear Regression: Tighter Error Bounds and Instance-Specific Uncertainty Estimationcited by 12
  12. 2021Variance Reduced Median-of-Means Estimator for Byzantine-Robust Distributed Inferencecited by 11
38 more
  1. 2021Bootstrapping the error of Oja's algorithmcited by 11
  2. 2022Statistical Inference of Constrained Stochastic Optimization via Sketched Sequential Quadratic Programmingcited by 10
  3. 2024High Confidence Level Inference is Almost Free using Parallel Stochastic Optimizationcited by 10
  4. 2025Gaussian Approximation and Multiplier Bootstrap for Stochastic Gradient Descentcited by 10
  5. 2023Acceleration of stochastic gradient descent with momentum by averaging: finite-sample rates and asymptotic normalitycited by 9
  6. 2022Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descentcited by 8
  7. 2023Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimalitycited by 8
  8. 2022Non-reversible Parallel Tempering for Deep Posterior Approximationcited by 7
  9. 2023Statistical Inference with Stochastic Gradient Methods under φ-mixing Datacited by 7
  10. 2023Online covariance estimation for stochastic gradient descent under Markovian samplingcited by 7
  11. 2021Statistical Inference for Polyak-Ruppert Averaged Zeroth-order Stochastic Gradient Algorithmcited by 6
  12. 2023Collapsed Inference for Bayesian Deep Learningcited by 6
  13. 2021Convergence and Optimality of Policy Gradient Methods in Weakly Smooth Settingscited by 5
  14. 2022Covariance Estimators for the ROOT-SGD Algorithm in Online Learningcited by 5
  15. 2023Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inferencecited by 5
  16. 2022Tuning Stochastic Gradient Algorithms for Statistical Inference via Large-Sample Asymptoticscited by 4
  17. 2023Fairness Uncertainty Quantification: How certain are you that the model is fair?cited by 4
  18. 2023Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learningcited by 4
  19. 2024Stochastic Optimization Algorithms for Instrumental Variable Regression with Streaming Datacited by 4
  20. 2025Online Covariance Estimation in Nonsmooth Stochastic Approximationcited by 4
  21. 2025Improved Central Limit Theorem and Bootstrap Approximations for Linear Stochastic Approximationcited by 4
  22. 2023SGMM: Stochastic Approximation to Generalized Method of Momentscited by 3
  23. 2024Asymptotic Time-Uniform Inference for Parameters in Averaged Stochastic Approximationcited by 3
  24. 2025Beyond Sin-Squared Error: Linear-Time Entrywise Uncertainty Quantification for Streaming PCAcited by 3
  25. 2021Online Statistical Inference for Parameters Estimation with Linear-Equality Constraintscited by 2
  26. 2021Statistical Estimation and Inference via Local SGD in Federated Learningcited by 2
  27. 2021A Physics inspired Functional Operator for Model Uncertainty Quantification in the RKHScited by 2
  28. 2022Statistical inference with implicit SGD: proximal Robbins-Monro vs. Polyak-Ruppertcited by 2
  29. 2024Asymptotic Analysis of Sample-averaged Q-learningcited by 2
  30. 2025Statistical Inference for Online Algorithmscited by 2
  31. 2025Statistical Inference for Differentially Private Stochastic Gradient Descentcited by 2
  32. 2025Federated Learning of Quantile Inference under Local Differential Privacycited by 2
  33. 2026Optimism Stabilizes Thompson Sampling for Adaptive Inferencecited by 2
  34. 2026Statistical Inference for Stochastic Gradient Descent: Beyond Finite Variancecited by 2
  35. 2026A Polyak-Ruppert Central Limit Theorem for SA-Adam with Momentum and Non-Convergent Adaptive Preconditioningcited by 2
  36. 2026How Much is Left? LLMs Linearly Encode Their Remaining Output Lengthcited by 2
  37. 2023Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regressioncited by 1
  38. 2023Out-of-distribution Object Detection through Bayesian Uncertainty Estimationcited by 1

50 most cited of 71 All 71 (JSON)