mlsanityData to 14 Oct 2026

Approximation theory for 1-Lipschitz ResNets

arXiv 2505.12003 ↗First posted 17 May 2025cs.LG cs.NA math.NAcited by 7

Not on the map. Why?

Abstract (arXiv)

1-Lipschitz neural networks are fundamental for generative modelling, inverse problems, and robust classifiers. In this paper, we focus on 1-Lipschitz residual networks (ResNets) based on explicit Euler steps of negative gradient flows and study their approximation capabilities. Leveraging the Restricted Stone-Weierstrass Theorem, we first show that these 1-Lipschitz ResNets are dense in the set of scalar 1-Lipschitz functions on any compact domain when width and depth are allowed to grow. We also show that these networks can exactly represent scalar piecewise affine 1-Lipschitz functions. We then prove a stronger statement: by inserting norm-constrained linear maps between the residual blocks, the same density holds when the hidden width is fixed. Because every layer obeys simple norm constraints, the resulting models can be trained with off-the-shelf optimisers. This paper provides the first universal approximation guarantees for 1-Lipschitz ResNets, laying a rigorous foundation for their practical use.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 32 of its 43 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

32 of 43 references.

References · 32

  1. 2013Intriguing properties of neural networkscited by 5,155
  2. 2017Improved Training of Wasserstein GANscited by 2,235
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
  4. 2016Error bounds for approximations with deep ReLU networkscited by 574
  5. 2018Sorting out Lipschitz function approximationcited by 213
  6. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 198
  7. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 195
  8. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  9. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  10. 2020The Lipschitz Constant of Self-Attentioncited by 174
  11. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 166
  12. 2019Orthogonal Convolutional Neural Networkscited by 123
20 more
  1. 2017Approximating Continuous Functions by ReLU Nets of Minimal Widthcited by 122
  2. 2021PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equationscited by 106
  3. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  4. 2018Adversarial Regularizers in Inverse Problemscited by 103
  5. 2022Neural Conservation Laws: A Divergence-Free Perspectivecited by 70
  6. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 60
  7. 2023How Smooth Is Attention?cited by 44
  8. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 39
  9. 2020Convolutional Proximal Neural Networks and Plug-and-Play Algorithmscited by 36
  10. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 34
  11. 2022Optimization-Induced Graph Implicit Nonlinear Diffusioncited by 28
  12. 2018Deep Theory of Functional Connections: A New Method for Estimating the Solutions of PDEscited by 17
  13. 2016Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)cited by 16
  14. 20231-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustnesscited by 16
  15. 2022Approximation of Lipschitz Functions using Deep Spline Neural Networkscited by 15
  16. 2022Dynamical systems' based neural networkscited by 10
  17. 2023Designing Stable Neural Networks using Convex Analysis and ODEscited by 10
  18. 2024You KAN Do It in a Single Shot: Plug-and-Play Methods with Single-Instance Priorscited by 2
  19. 2025Approximation properties of neural ODEscited by 2
  20. 2023Resilient Graph Neural Networks: A Coupled Dynamical Systems Approachcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2026Approximation Theory for Lipschitz Continuous Transformerscited by 5
  2. 2025Learning where to learn: Training data distribution optimization for scientific machine learningcited by 3
  3. 2025Simultaneously Solving Infinitely Many LQ Mean Field Games In Hilbert Spaces: The Power of Neural Operatorscited by 3
  4. 2025Expressive Power of Implicit Models: Rich Equilibria and Test-Time Scalingcited by 1
  5. 2025Deep Network Trainability via Persistent Subspace Orthogonalitycited by 0
  6. 2026Approximation of Maximally Monotone Operators : A Graph Convergence Perspectivecited by 0
  7. 20261-Lipschitz Neural Networks on Hadamard Manifoldscited by 0