mlsanityData to 7 Oct 2026

Iterate averaging as regularization for stochastic gradient descent

arXiv 1802.08009 ↗First posted 22 Feb 2018cs.LG stat.MLcited by 27

Not on the map. Why?

Abstract (arXiv)

We propose and analyze a variant of the classic Polyak-Ruppert averaging scheme, broadly used in stochastic gradient methods. Rather than a uniform average of the iterates, we consider a weighted average, with weights decaying in a geometric fashion. In the context of linear least squares regression, we show that this averaging scheme has a the same regularizing effect, and indeed is asymptotically equivalent, to ridge regression. In particular, we derive finite-sample bounds for the proposed approach that match the best known results for regularized stochastic gradient methods.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 4; 2023: 7; 2024: 3; 2025: 2; 2026: 3.

27 papers.

Sort:

By area:

  1. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  2. 2023Deep Model Fusion: A Surveycited by 89
  3. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  4. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 65
  5. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  6. 2021Last Iterate Risk Bounds of SGD with Decaying Stepsize for Overparameterized Linear Regressioncited by 37
  7. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 36
  8. 2021Implicit Sparse Regularization: The Impact of Depth and Early Stoppingcited by 30
  9. 2022Stochastic Weight Averaging Revisitedcited by 26
  10. 2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
  11. 2021Eigencurve: Optimal Learning Rate Schedule for SGD on Quadratic Objectives with Skewed Hessian Spectrumscited by 15
  12. 2023Average of Pruning: Improving Performance and Stability of Out-of-Distribution Detectioncited by 14
15 more
  1. 2023Generalized equivalences between subsampling and ridge regularizationcited by 10
  2. 2022Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regimecited by 8
  3. 2023Convergence and concentration properties of constant step-size SGD through Markov chainscited by 8
  4. 2023Inconsistency, Instability, and Generalization Gap of Deep Neural Network Trainingcited by 6
  5. 2025When, Where and Why to Average Weights?cited by 6
  6. 2021Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models?cited by 5
  7. 2024Failures and Successes of Cross-Validation for Early-Stopped Gradient Descentcited by 4
  8. 2025Design Considerations in Offline Preference-based RLcited by 4
  9. 2021Server Averaging for Federated Learningcited by 2
  10. 2024Implicit Regularization Paths of Weighted Neural Representationscited by 1
  11. 2026Generalization in Nonlinear Least Squares via Learned Feature Geometrycited by 1
  12. 2022Variational Factorization Machines for Preference Elicitation in Large-Scale Recommender Systemscited by 0
  13. 2023First-order ANIL provably learns representations despite overparametrizationcited by 0
  14. 2026On-Average Stability of Multipass Preconditioned SGD and Effective Dimensioncited by 0
  15. 2026Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretrainingcited by 0