# Direct Parameterization of Lipschitz-Bounded Deep Networks

arXiv: https://arxiv.org/abs/2301.11526 · HTML: https://mlsanity.com/paper/2301.11526/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 73 references that are in the map.
- First posted on arXiv 27 Jan 2023.

## Builds on (most cited first, 30 of 38)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks](https://mlsanity.com/paper/2003.01690/) (2020)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Provable defenses against adversarial examples via the convex outer adversarial polytope](https://mlsanity.com/paper/1711.00851/) (2017)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Models](https://mlsanity.com/paper/1810.12715/) (2018)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Certified Defenses against Adversarial Examples](https://mlsanity.com/paper/1801.09344/) (2018)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Safe Control with Learned Certificates: A Survey of Neural Lyapunov, Barrier, and Contraction methods](https://mlsanity.com/paper/2202.11762/) (2022)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Orthogonalizing Convolutional Layers with the Cayley Transform](https://mlsanity.com/paper/2104.07167/) (2021)
- [Monotone operator equilibrium networks](https://mlsanity.com/paper/2006.08591/) (2020)
- [Training robust neural networks using Lipschitz bounds](https://mlsanity.com/paper/2005.02929/) (2020)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Learning Smooth Neural Functions via Lipschitz Regularization](https://mlsanity.com/paper/2202.08345/) (2022)
- [Generalizable Adversarial Training via Spectral Normalization](https://mlsanity.com/paper/1811.07457/) (2018)
- [Skew Orthogonal Convolutions](https://mlsanity.com/paper/2105.11417/) (2021)
- [A Dynamical System Perspective for Lipschitz Neural Networks](https://mlsanity.com/paper/2110.12690/) (2021)
- [Lipschitz Bounded Equilibrium Networks](https://mlsanity.com/paper/2010.01732/) (2020)
- [Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100](https://mlsanity.com/paper/2108.04062/) (2021)
- [Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustness](https://mlsanity.com/paper/2104.05942/) (2021)

## Built on by (most cited first, 30 of 51)

- [Some Fundamental Aspects about Lipschitz Continuity of Neural Networks](https://mlsanity.com/paper/2302.10886/) (2023)
- [Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularization](https://mlsanity.com/paper/2310.00116/) (2023)
- [1-Lipschitz Layers Compared: Memory, Speed, and Certifiable Robustness](https://mlsanity.com/paper/2311.16833/) (2023)
- [LipSim: A Provably Robust Perceptual Similarity Metric](https://mlsanity.com/paper/2310.18274/) (2023)
- [Smoothness Really Matters: A Simple Yet Effective Approach for Unsupervised Graph Domain Adaptation](https://mlsanity.com/paper/2412.11654/) (2024)
- [DP-SGD Without Clipping: The Lipschitz Neural Network Way](https://mlsanity.com/paper/2305.16202/) (2023)
- [ECLipsE: Efficient Compositional Lipschitz Constant Estimation for Deep Neural Networks](https://mlsanity.com/paper/2404.04375/) (2024)
- [Lipschitz-bounded 1D convolutional neural networks using the Cayley transform and the controllability Gramian](https://mlsanity.com/paper/2303.11835/) (2023)
- [Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networks](https://mlsanity.com/paper/2402.01344/) (2024)
- [The Lipschitz-Variance-Margin Tradeoff for Enhanced Randomized Smoothing](https://mlsanity.com/paper/2309.16883/) (2023)
- [Lipschitz constant estimation for general neural network architectures using control tools](https://mlsanity.com/paper/2405.01125/) (2024)
- [An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures](https://mlsanity.com/paper/2501.07930/) (2025)
- [Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations](https://mlsanity.com/paper/2401.14033/) (2024)
- [Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Loss](https://mlsanity.com/paper/2505.15174/) (2025)
- [A Recipe for Improved Certifiable Robustness](https://mlsanity.com/paper/2310.02513/) (2023)
- [On Robust Reinforcement Learning with Lipschitz-Bounded Policy Networks](https://mlsanity.com/paper/2405.11432/) (2024)
- [Compositional Curvature Bounds for Deep Neural Networks](https://mlsanity.com/paper/2406.05119/) (2024)
- [Statistically Optimal Generative Modeling with Maximum Deviation from the Empirical Distribution](https://mlsanity.com/paper/2307.16422/) (2023)
- [Certified Robust Models with Slack Control and Large Lipschitz Constants](https://mlsanity.com/paper/2309.06166/) (2023)
- [LipShiFT: A Certifiably Robust Shift-based Vision Transformer](https://mlsanity.com/paper/2503.14751/) (2025)
- [Learning conditional distributions on continuous spaces](https://mlsanity.com/paper/2406.09375/) (2024)
- [A Complete Set of Quadratic Constraints for Repeated ReLU and Generalizations](https://mlsanity.com/paper/2407.06888/) (2024)
- [LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layers](https://mlsanity.com/paper/2410.22258/) (2024)
- [Improved Scalable Lipschitz Bounds for Deep Neural Networks](https://mlsanity.com/paper/2503.14297/) (2025)
- [R2DN: Scalable Parameterization of Contracting and Lipschitz Recurrent Deep Networks](https://mlsanity.com/paper/2504.01250/) (2025)
- [Structured Transformations for Stable and Interpretable Neural Computation](https://mlsanity.com/paper/2508.00127/) (2025)
- [LipNeXt: Scaling up Lipschitz-based Certified Robustness to Billion-parameter Models](https://mlsanity.com/paper/2601.18513/) (2026)
- [Learning Stable and Robust Linear Parameter-Varying State-Space Models](https://mlsanity.com/paper/2304.01828/) (2023)
- [RobustNeuralNetworks.jl: a Package for Machine Learning and Data-Driven Control with Certified Robustness](https://mlsanity.com/paper/2306.12612/) (2023)
- [Automatic Optimisation of Normalised Neural Networks](https://mlsanity.com/paper/2312.10672/) (2023)

## Abstract (arXiv)

This paper introduces a new parameterization of deep neural networks (both fully-connected and convolutional) with guaranteed ℓ2 Lipschitz bounds, i.e. limited sensitivity to input perturbations. The Lipschitz guarantees are equivalent to the tightest-known bounds based on certification via a semidefinite program (SDP). We provide a ``direct'' parameterization, i.e., a smooth mapping from mathbb RN onto the set of weights satisfying the SDP-based bound. Moreover, our parameterization is complete, i.e. a neural network satisfies the SDP bound if and only if it can be represented via our parameterization. This enables training using standard gradient methods, without any inner approximation or computationally intensive tasks (e.g. projections or barrier terms) for the SDP constraint. The new parameterization can equivalently be thought of as either a new layer type (the textitsandwich layer), or a novel parameterization of standard feedforward networks with parameter sharing between neighbouring layers. A comprehensive set of experiments on image classification shows that sandwich layers outperform previous approaches on both empirical and certified robust accuracy. Code is available at urlhttps://github.com/acfr/LBDN.