mlsanityData to 14 Oct 2026

Optimization-Induced Dynamics of Lipschitz Continuity in Neural Networks

arXiv 2506.18588 ↗First posted 23 Jun 2025cs.LG cs.AI stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

Lipschitz continuity characterizes the worst-case sensitivity of neural networks to small input perturbations; yet its dynamics (i.e. temporal evolution) during training remains under-explored. We present a rigorous mathematical framework to model the temporal evolution of Lipschitz continuity during training with stochastic gradient descent (SGD). This framework leverages a system of stochastic differential equations (SDEs) to capture both deterministic and stochastic forces. Our theoretical analysis identifies three principal factors driving the evolution: (i) the projection of gradient flows, induced by the optimization dynamics, onto the operator-norm Jacobian of parameter matrices; (ii) the projection of gradient noise, arising from the randomness in mini-batch sampling, onto the operator-norm Jacobian; and (iii) the projection of the gradient noise onto the operator-norm Hessian of parameter matrices. Furthermore, our theoretical framework sheds light on such as how noisy supervision, parameter initialization, batch size, and mini-batch sampling trajectories, among other factors, shape the evolution of the Lipschitz continuity of neural networks. Our experimental results demonstrate strong agreement between the theoretical implications and the observed behaviors.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 38 of its 69 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

38 of 69 references.

References · 38

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  3. 2013Intriguing properties of neural networkscited by 5,155
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  8. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 665
  9. 2014New insights and perspectives on the natural gradient methodcited by 443
  10. 2017Robust Loss Functions under Label Noise for Deep Neural Networkscited by 443
  11. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  12. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
26 more
  1. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  2. 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
  3. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 293
  4. 2017Three Factors Influencing Minima in SGDcited by 273
  5. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  6. 2015A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insightscited by 206
  7. 2017DART: Noise Injection for Robust Imitation Learningcited by 191
  8. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  9. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  10. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 145
  11. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  12. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  13. 2018Stochastic Modified Equations and Dynamics of Stochastic Gradient Algorithms I: Mathematical Foundationscited by 114
  14. 2017Stochastic gradient descent performs variational inference, converges to limit cycles for deep networkscited by 109
  15. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  16. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  17. 2018The Singular Values of Convolutional Layerscited by 95
  18. 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
  19. 2018A Walk with SGDcited by 78
  20. 2022Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspectivecited by 51
  21. 2020Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noisecited by 33
  22. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 26
  23. 2023Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularizationcited by 21
  24. 2023On the Lipschitz Constant of Deep Networks and Double Descentcited by 11
  25. 2024Interpreting Global Perturbation Robustness of Image Models using Axiomatic Spectral Importance Decompositioncited by 1
  26. 2025Higher-Order Singular-Value Derivatives of Rectangular Real Matricescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026A Stochastic--Geometric Theory of Scaling Laws in Grokkingcited by 0
  2. 2026Lipschitz Continuity in Deep Learning: A Systematic Review of Theoretical Foundations, Estimation Methods, Regularization Approaches, and Certifiable Robustnesscited by 0