mlsanityData to 7 Oct 2026

Weighted Averaged Stochastic Gradient Descent: Asymptotic Normality and Optimality

arXiv 2307.06915 ↗First posted 13 Jul 2023stat.ML cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Stochastic Gradient Descent (SGD) is one of the most popular algorithms in statistical and machine learning due to its computational and memory efficiency. Various averaging schemes have been proposed to accelerate the convergence of SGD in different settings. In this paper, we explore a general averaging scheme for SGD. Specifically, we establish the asymptotic normality of a broad range of weighted averaged SGD solutions and provide asymptotically valid online inference approaches. Furthermore, we propose an adaptive averaging scheme that exhibits both optimal statistical rate and favorable non-asymptotic convergence, drawing insights from the optimal weight for the linear model in terms of non-asymptotic mean squared error (MSE).

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 15 of its 38 references that are in the map. First posted on arXiv 13 Jul 2023.

Builds on

15 of 38 references.

References · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  3. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  4. 2013Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithmcited by 132
  5. 2018Tight Analyses for Non-Smooth Stochastic Gradient Descentcited by 94
  6. 2014Deep learning with Elastic Averaging SGDcited by 90
  7. 2016Statistical Inference for Model Parameters in Stochastic Gradient Descentcited by 71
  8. 2020On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentrationcited by 65
  9. 2020An Analysis of Constant Step Size SGD in the Non-convex Regime: Asymptotic Normality and Biascited by 45
  10. 2020Online Covariance Matrix Estimation in Stochastic Gradient Descentcited by 37
  11. 2019Normal Approximation for Stochastic Gradient Descent via Non-Asymptotic Rates of Martingale CLTcited by 30
  12. 2021Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scalingcited by 28
3 more
  1. 2017Statistical inference using SGDcited by 21
  2. 2020Statistical Inference for Online Decision Making via Stochastic Gradient Descentcited by 17
  3. 2017Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradientscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2024High Confidence Level Inference is Almost Free using Parallel Stochastic Optimizationcited by 9
  2. 2025Statistical Inference for Online Algorithmscited by 2
  3. 2025Sharp Gaussian approximations for Decentralized Federated Learningcited by 1
  4. 2024Enhancing Stochastic Optimization for Statistical Efficiency Using ROOT-SGD with Diminishing Stepsizecited by 0
  5. 2025Online Statistical Inference of Constrained Stochastic Optimization via Random Scalingcited by 0
  6. 2026Sharp asymptotic theory for Q-learning with LDTZ learning rate and its generalizationcited by 0
  7. 2026A Polyak-Ruppert Central Limit Theorem for SA-Adam with Momentum and Non-Convergent Adaptive Preconditioningcited by 0