mlsanityData to 14 Oct 2026

Designing Stable Neural Networks using Convex Analysis and ODEs

arXiv 2306.17332 ↗First posted 29 Jun 2023cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further analysis of the proposed architecture shows that the spectral norms of the weights can be further constrained to ensure that the network is an averaged operator, making it a natural candidate for a learned denoiser in Plug-and-Play algorithms. Using a novel adaptive way of enforcing the spectral norm constraints, we show that, even with these constraints, it is possible to train performant networks. The proposed architecture is applied to the problem of adversarially robust image classification, to image denoising, and finally to the inverse problem of deblurring.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 29 Jun 2023.

Builds on

26 of 47 references.

References · 26

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  4. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  5. 2018Neural Ordinary Differential Equationscited by 3,779
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2016Wide Residual Networkscited by 3,070
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  10. 2016Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoisingcited by 1,358
  11. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  12. 2016Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applicationscited by 246
14 more
  1. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 198
  2. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 195
  3. 2018Deep Neural Networks Motivated by Partial Differential Equationscited by 187
  4. 2015Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolationcited by 78
  5. 2017Reversible Architectures for Arbitrarily Deep Residual Neural Networkscited by 70
  6. 2018An Online Plug-and-Play Algorithm for Regularized Image Reconstructioncited by 70
  7. 2020Learning Maximally Monotone Operators for Image Recoverycited by 62
  8. 2020Convolutional Proximal Neural Networks and Plug-and-Play Algorithmscited by 36
  9. 2020Structure preserving deep learningcited by 31
  10. 2018Deep Neural Network Structures Solving Variational Inequalitiescited by 26
  11. 2022Dynamical systems' based neural networkscited by 10
  12. 2019Contractivity of Runge-Kutta methods for convex gradient systemscited by 4
  13. 2016Algorithm-Induced Prior for Image Restorationcited by 2
  14. 2022Deep neural networks on diffeomorphism groups for optimal shape reparameterizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 4; 2026: 2.

10 papers.

  1. 2025Approximation theory for 1-Lipschitz ResNetscited by 7
  2. 2026Approximation Theory for Lipschitz Continuous Transformerscited by 5
  3. 2024Symplectic Neural Networks Based on Dynamical Systemscited by 4
  4. 2023From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networkscited by 3
  5. 2024Invertible ResNets for Inverse Imaging Problems: Competitive Performance with Provable Regularization Propertiescited by 2
  6. 2023Resilient Graph Neural Networks: A Coupled Dynamical Systems Approachcited by 1
  7. 2025Understanding Learning Dynamics Through Structured Representationscited by 1
  8. 2025Deep Network Trainability via Persistent Subspace Orthogonalitycited by 0
  9. 2025Stable neural networks and connections to continuous dynamical systemscited by 0
  10. 20261-Lipschitz Neural Networks on Hadamard Manifoldscited by 0