mlsanityData to 14 Oct 2026

Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

arXiv 2106.03156 ↗First posted 6 Jun 2021stat.ML cs.LG econ.EMcited by 28

Not on the map. Why?

Abstract (arXiv)

We develop a new method of online inference for a vector of parameters estimated by the Polyak-Ruppert averaging procedure of stochastic gradient descent (SGD) algorithms. We leverage insights from time series regression in econometrics and construct asymptotically pivotal statistics via random scaling. Our approach is fully operational with online data and is rigorously underpinned by a functional central limit theorem. Our proposed inference method has a couple of key advantages over the existing methods. First, the test statistic is computed in an online fashion with only SGD iterates and the critical values can be obtained without any resampling methods, thereby allowing for efficient implementation suitable for massive online data. Second, there is no need to estimate the asymptotic variance and our inference method is shown to be robust to changes in the tuning parameters for SGD algorithms in simulation experiments with synthetic data.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 9 of its 36 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

9 of 36 references.

References · 9

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 111
  3. 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
  4. 2020On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentrationcited by 68
  5. 2020Online Covariance Matrix Estimation in Stochastic Gradient Descentcited by 37
  6. 2019Normal Approximation for Stochastic Gradient Descent via Non-Asymptotic Rates of Martingale CLTcited by 30
  7. 2021Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methodscited by 14
  8. 2019On Constructing Confidence Region for Model Parameters in Stochastic Gradient Descent via Batch Meanscited by 13
  9. 2017Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradientscited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 7; 2024: 4; 2025: 5; 2026: 8.

28 papers.

Sort:

By area:

  1. 2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
  2. 2023Online Statistical Inference for Nonlinear Stochastic Approximation with Markovian Datacited by 20
  3. 2024High Confidence Level Inference is Almost Free using Parallel Stochastic Optimizationcited by 10
  4. 2022Online Statistical Inference for Contextual Bandits via Stochastic Gradient Descentcited by 8
  5. 2023Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimalitycited by 8
  6. 2023Statistical Inference with Stochastic Gradient Methods under φ-mixing Datacited by 7
  7. 2022Covariance Estimators for the ROOT-SGD Algorithm in Online Learningcited by 5
  8. 2023Effectiveness of Constant Stepsize in Markovian LSA and Statistical Inferencecited by 5
  9. 2023SGMM: Stochastic Approximation to Generalized Method of Momentscited by 3
  10. 2024Asymptotic Time-Uniform Inference for Parameters in Averaged Stochastic Approximationcited by 3
  11. 2025Beyond Sin-Squared Error: Linear-Time Entrywise Uncertainty Quantification for Streaming PCAcited by 3
  12. 2021Statistical Estimation and Inference via Local SGD in Federated Learningcited by 2
16 more
  1. 2024Asymptotic Analysis of Sample-averaged Q-learningcited by 2
  2. 2025Statistical Inference for Online Algorithmscited by 2
  3. 2025Statistical Inference for Differentially Private Stochastic Gradient Descentcited by 2
  4. 2026Statistical Inference for Stochastic Gradient Descent: Beyond Finite Variancecited by 2
  5. 2026A Polyak-Ruppert Central Limit Theorem for SA-Adam with Momentum and Non-Convergent Adaptive Preconditioningcited by 2
  6. 2026How Much is Left? LLMs Linearly Encode Their Remaining Output Lengthcited by 2
  7. 2023Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regressioncited by 1
  8. 2025Statistical Inference for Linear Functionals of Online Least-squares SGD when t gtrsim d1+δcited by 1
  9. 2026Self-Normalized Inference for Constant-Stepsize Temporal-Difference Learning under Markovian Samplingcited by 1
  10. 2023Cheap Bootstrap for Fast Uncertainty Quantification of Stochastic Gradient Descentcited by 0
  11. 2024Enhancing Stochastic Optimization for Statistical Efficiency Using ROOT-SGD with Diminishing Stepsizecited by 0
  12. 2025Online Statistical Inference of Constrained Stochastic Optimization via Random Scalingcited by 0
  13. 2026Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalizationcited by 0
  14. 2026Online Covariance Estimation in Averaged SGD: Improved Batch-Mean Rates and Minimax Optimality via Trajectory Regressioncited by 0
  15. 2026Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reductioncited by 0
  16. 2026SAUSS: Stochastic Approximation with Unbiased Simulated Scores for Limited Dependent Variable Modelscited by 0