mlsanityData to 14 Oct 2026

Convergence Analysis for Training Stochastic Neural Networks via Stochastic Gradient Descent

arXiv 2212.08924 ↗First posted 17 Dec 2022math.NA cs.LG cs.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we carry out numerical analysis to prove convergence of a novel sample-wise back-propagation method for training a class of stochastic neural networks (SNNs). The structure of the SNN is formulated as discretization of a stochastic differential equation (SDE). A stochastic optimal control framework is introduced to model the training procedure, and a sample-wise approximation scheme for the adjoint backward SDE is applied to improve the efficiency of the stochastic optimal control solver, which is equivalent to the back-propagation for training the SNN. The convergence analysis is derived with and without convexity assumption for optimization of the SNN parameters. Especially, our analysis indicates that the number of SNN training steps should be proportional to the square of the number of layers in the convex optimization case. Numerical experiments are carried out to validate the analysis results, and the performance of the sample-wise back-propagation method for training SNNs is examined by benchmark machine learning examples.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 45 references that are in the map. First posted on arXiv 17 Dec 2022.

Builds on

19 of 45 references.

References · 19

  1. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2018Neural Ordinary Differential Equationscited by 3,779
  4. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  5. 2017Efficient Processing of Deep Neural Networks: A Tutorial and Surveycited by 434
  6. 2019Augmented Neural ODEscited by 397
  7. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  8. 2017Stable Architectures for Deep Neural Networkscited by 341
  9. 2020B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Datacited by 335
  10. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
  11. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
  12. 2020Scalable Gradients for Stochastic Differential Equationscited by 271
7 more
  1. 2013OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networkscited by 224
  2. 2019Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limitcited by 176
  3. 2019Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networkscited by 129
  4. 2020SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimatescited by 75
  5. 2019Quality of Uncertainty Quantification for Bayesian Neural Network Inferencecited by 59
  6. 2018Quantifying model form uncertainty in Reynolds-averaged turbulence models with Bayesian deep neural networkscited by 20
  7. 2017Bayesian Recurrent Neural Network Models for Forecasting and Quantifying Uncertainty in Spatial-Temporal Datacited by 17

Built on by

Not cited yet.