# Designing Stable Neural Networks using Convex Analysis and ODEs

arXiv: https://arxiv.org/abs/2306.17332 · HTML: https://mlsanity.com/paper/2306.17332/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 29 Jun 2023.

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Plug-and-Play ADMM for Image Restoration: Fixed Point Convergence and Applications](https://mlsanity.com/paper/1605.01710/) (2016)
- [Plug-and-Play Methods Provably Converge with Properly Trained Denoisers](https://mlsanity.com/paper/1905.05406/) (2019)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation](https://mlsanity.com/paper/1512.07331/) (2015)
- [Reversible Architectures for Arbitrarily Deep Residual Neural Networks](https://mlsanity.com/paper/1709.03698/) (2017)
- [An Online Plug-and-Play Algorithm for Regularized Image Reconstruction](https://mlsanity.com/paper/1809.04693/) (2018)
- [Learning Maximally Monotone Operators for Image Recovery](https://mlsanity.com/paper/2012.13247/) (2020)
- [Convolutional Proximal Neural Networks and Plug-and-Play Algorithms](https://mlsanity.com/paper/2011.02281/) (2020)
- [Structure preserving deep learning](https://mlsanity.com/paper/2006.03364/) (2020)
- [Deep Neural Network Structures Solving Variational Inequalities](https://mlsanity.com/paper/1808.07526/) (2018)
- [Dynamical systems' based neural networks](https://mlsanity.com/paper/2210.02373/) (2022)
- [Contractivity of Runge-Kutta methods for convex gradient systems](https://mlsanity.com/paper/1909.09971/) (2019)
- [Algorithm-Induced Prior for Image Restoration](https://mlsanity.com/paper/1602.00715/) (2016)
- [Deep neural networks on diffeomorphism groups for optimal shape reparameterization](https://mlsanity.com/paper/2207.11141/) (2022)

## Built on by (most cited first, 10 of 10)

- [Approximation theory for 1-Lipschitz ResNets](https://mlsanity.com/paper/2505.12003/) (2025)
- [Approximation Theory for Lipschitz Continuous Transformers](https://mlsanity.com/paper/2602.15503/) (2026)
- [Symplectic Neural Networks Based on Dynamical Systems](https://mlsanity.com/paper/2408.09821/) (2024)
- [From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks](https://mlsanity.com/paper/2307.02279/) (2023)
- [Invertible ResNets for Inverse Imaging Problems: Competitive Performance with Provable Regularization Properties](https://mlsanity.com/paper/2409.13482/) (2024)
- [Resilient Graph Neural Networks: A Coupled Dynamical Systems Approach](https://mlsanity.com/paper/2311.06942/) (2023)
- [Understanding Learning Dynamics Through Structured Representations](https://mlsanity.com/paper/2508.02126/) (2025)
- [Deep Network Trainability via Persistent Subspace Orthogonality](https://mlsanity.com/paper/2508.02882/) (2025)
- [Stable neural networks and connections to continuous dynamical systems](https://mlsanity.com/paper/2510.22299/) (2025)
- [1-Lipschitz Neural Networks on Hadamard Manifolds](https://mlsanity.com/paper/2607.19335/) (2026)

## Abstract (arXiv)

Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further analysis of the proposed architecture shows that the spectral norms of the weights can be further constrained to ensure that the network is an averaged operator, making it a natural candidate for a learned denoiser in Plug-and-Play algorithms. Using a novel adaptive way of enforcing the spectral norm constraints, we show that, even with these constraints, it is possible to train performant networks. The proposed architecture is applied to the problem of adversarially robust image classification, to image denoising, and finally to the inverse problem of deblurring.