mlsanityData to 7 Oct 2026

Perturbed Iterate SGD for Lipschitz Continuous Loss Functions with Numerical Error and Adaptive Step Sizes

arXiv 2211.04655 ↗First posted 9 Nov 2022math.OC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Motivated by neural network training in finite-precision arithmetic environments, this work studies the convergence of perturbed iterate SGD using adaptive step sizes in an environment with numerical error. Considering a general stochastic Lipschitz continuous loss function, an asymptotic convergence result to a Clarke stationary point is proven as well as the non-asymptotic convergence to an approximate stationary point in expectation. It is assumed that only an approximation of the loss function's stochastic gradient can be computed, in addition to error in computing the SGD step itself.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 50 references that are in the map. First posted on arXiv 9 Nov 2022.

Builds on

19 of 50 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  4. 2015Deep Learning with Limited Numerical Precisioncited by 368
  5. 2019Why gradient clipping accelerates training: A theoretical justification for adaptivitycited by 335
  6. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  7. 2018Stochastic subgradient method converges on tame functionscited by 120
  8. 2020Improved Analysis of Clipping Algorithms for Non-convex Optimizationcited by 88
  9. 2020A mathematical model for automatic differentiation in machine learningcited by 44
  10. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 40
  11. 2021A gradient sampling method with complexity guarantees for Lipschitz functions in high and low dimensionscited by 37
  12. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
7 more
  1. 2021On the Finite-Time Complexity and Practical Computation of Approximate Stationarity Concepts of Lipschitz Functionscited by 30
  2. 2019QPyTorch: A Low-Precision Arithmetic Simulation Frameworkcited by 21
  3. 2020NITI: Training Integer Neural Networks Using Integer-only Arithmeticcited by 18
  4. 2017Energy-efficient Amortized Inference with Cascaded Deep Classifierscited by 15
  5. 2018Testing Halfspaces over Rotation-Invariant Distributionscited by 2
  6. 2020Perturbed Iterate SGD for Lipschitz Continuous Loss Functionscited by 2
  7. 2022Sparse Training with Lipschitz Continuous Loss Functions and a Weighted Group L0-norm Constraintcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Mathematical Challenges in Deep Learningcited by 0