mlsanityData to 7 Oct 2026

Stochastic Gradient Langevin Dynamics Based on Quantization with Increasing Resolution

arXiv 2305.18864 ↗First posted 30 May 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Stochastic learning dynamics based on Langevin or Levy stochastic differential equations (SDEs) in deep neural networks control the variance of noise by varying the size of the mini-batch or directly those of injecting noise. Since the noise variance affects the approximation performance, the design of the additive noise is significant in SDE-based learning and practical implementation. In this paper, we propose an alternative stochastic descent learning equation based on quantized optimization for non-convex objective functions, adopting a stochastic analysis perspective. The proposed method employs a quantized optimization approach that utilizes Langevin SDE dynamics, allowing for controllable noise with an identical distribution without the need for additive noise or adjusting the mini-batch size. Numerical experiments demonstrate the effectiveness of the proposed algorithm on vanilla convolution neural network(CNN) models and the ResNet-50 architecture across various data sets. Furthermore, we provide a simple PyTorch implementation of the proposed algorithm.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 64 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

33 of 64 references.

References · 33

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  5. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  6. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  7. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 298
  8. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  9. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  10. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  11. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 213
  12. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
21 more
  1. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 190
  2. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  3. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  4. 2017User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradientcited by 116
  5. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  6. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 109
  7. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 108
  8. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  9. 2017Global Convergence of Langevin Dynamics Based Algorithms for Nonconvex Optimizationcited by 96
  10. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  11. 2020On the Generalization Benefit of Noise in Stochastic Gradient Descentcited by 81
  12. 2021On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)cited by 73
  13. 2022Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Losscited by 60
  14. 2018The promises and pitfalls of Stochastic Gradient Langevin Dynamicscited by 56
  15. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50
  16. 2018On the insufficiency of existing momentum schemes for Stochastic Optimizationcited by 46
  17. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 43
  18. 2015Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithmscited by 31
  19. 2022Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Functioncited by 19
  20. 2022Low-Precision Stochastic Gradient Langevin Dynamicscited by 12
  21. 2022Statistical Learning and Inverse Problems: A Stochastic Gradient Approachcited by 5

Built on by

Not cited yet.