# CLIP: Cheap Lipschitz Training of Neural Networks

arXiv: https://arxiv.org/abs/2103.12531 · HTML: https://mlsanity.com/paper/2103.12531/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 30 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 22 of 22)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Adversarial Training for Free!](https://mlsanity.com/paper/1904.12843/) (2019)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Lipschitz Certificates for Layered Network Structures Driven by Averaged Activation Operators](https://mlsanity.com/paper/1903.01014/) (2019)
- [Limitations of the Lipschitz constant as a defense against adversarial examples](https://mlsanity.com/paper/1807.09705/) (2018)
- [Adversarial Lipschitz Regularization](https://mlsanity.com/paper/1907.05681/) (2019)
- [On Lipschitz Bounds of General Convolutional Neural Networks](https://mlsanity.com/paper/1808.01415/) (2018)
- [Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant](https://mlsanity.com/paper/2001.06263/) (2020)
- [Lipschitz regularized Deep Neural Networks generalize and are adversarially robust](https://mlsanity.com/paper/1808.09540/) (2018)
- [Lipschitz Bounds and Provably Robust Training by Laplacian Smoothing](https://mlsanity.com/paper/2006.03712/) (2020)
- [Large Norms of CNN Layers Do Not Hurt Adversarial Robustness](https://mlsanity.com/paper/2009.08435/) (2020)
- [Variational regularisation for inverse problems with imperfect forward operators and general noise models](https://mlsanity.com/paper/2005.14131/) (2020)
- [Solution Paths of Variational Regularization Methods for Inverse Problems](https://mlsanity.com/paper/1808.01783/) (2018)
- [Towards Rapid and Robust Adversarial Training with One-Step Attacks](https://mlsanity.com/paper/2002.10097/) (2020)

## Built on by (most cited first, 29 of 29)

- [Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space](https://mlsanity.com/paper/2402.09063/) (2024)
- [Exploring Misclassifications of Robust Neural Networks to Enhance Adversarial Attacks](https://mlsanity.com/paper/2105.10304/) (2021)
- [Some Fundamental Aspects about Lipschitz Continuity of Neural Networks](https://mlsanity.com/paper/2302.10886/) (2023)
- [Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions](https://mlsanity.com/paper/2210.16222/) (2022)
- [The Geometry of Adversarial Training in Binary Classification](https://mlsanity.com/paper/2111.13613/) (2021)
- [A Quantitative Geometric Approach to Neural-Network Smoothness](https://mlsanity.com/paper/2203.01212/) (2022)
- [Improving Robustness against Real-World and Worst-Case Distribution Shifts through Decision Region Quantification](https://mlsanity.com/paper/2205.09619/) (2022)
- [Approximation of Lipschitz Functions using Deep Spline Neural Networks](https://mlsanity.com/paper/2204.06233/) (2022)
- [Connections between Numerical Algorithms for PDEs and Neural Networks](https://mlsanity.com/paper/2107.14742/) (2021)
- [Memory-efficient model-based deep learning with convergence and robustness guarantees](https://mlsanity.com/paper/2206.04797/) (2022)
- [Raising the Bar for Certified Adversarial Robustness with Diffusion Models](https://mlsanity.com/paper/2305.10388/) (2023)
- [Consistency of Neural Causal Partial Identification](https://mlsanity.com/paper/2405.15673/) (2024)
- [Approximation Theory for Lipschitz Continuous Transformers](https://mlsanity.com/paper/2602.15503/) (2026)
- [Learning Globally Smooth Functions on Manifolds](https://mlsanity.com/paper/2210.00301/) (2022)
- [Policy Verification in Stochastic Dynamical Systems Using Logarithmic Neural Certificates](https://mlsanity.com/paper/2406.00826/) (2024)
- [Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods](https://mlsanity.com/paper/2108.13993/) (2021)
- [Invertible ResNets for Inverse Imaging Problems: Competitive Performance with Provable Regularization Properties](https://mlsanity.com/paper/2409.13482/) (2024)
- [A Tunable Despeckling Neural Network Stabilized via Diffusion Equation](https://mlsanity.com/paper/2411.15921/) (2024)
- [Effective Data Pruning through Score Extrapolation](https://mlsanity.com/paper/2506.09010/) (2025)
- [Consensus-based optimization for closed-box adversarial attacks and a connection to evolution strategies](https://mlsanity.com/paper/2506.24048/) (2025)
- [Error Analysis of Bayesian Inverse Problems with Generative Priors](https://mlsanity.com/paper/2601.17374/) (2026)
- [Local monotone operator learning using non-monotone operators: MnM-MOL](https://mlsanity.com/paper/2312.00386/) (2023)
- [Learning Lipschitz Feedback Policies from Expert Demonstrations: Closed-Loop Guarantees, Generalization and Robustness](https://mlsanity.com/paper/2103.16629/) (2021)
- [Plug-and-Play Deep Energy Model for Inverse problems](https://mlsanity.com/paper/2302.11570/) (2023)
- [Accelerated parallel MRI using memory efficient and robust monotone operator learning (MOL)](https://mlsanity.com/paper/2304.01351/) (2023)
- [Your Network May Need to Be Rewritten: Network Adversarial Based on High-Dimensional Function Graph Decomposition](https://mlsanity.com/paper/2405.03712/) (2024)
- [MAP Image Recovery with Guarantees using Locally Convex Multi-Scale Energy (LC-MUSE) Model](https://mlsanity.com/paper/2502.03302/) (2025)
- [Deep End-to-End Posterior ENergy (DEEPEN) for image recovery](https://mlsanity.com/paper/2503.17244/) (2025)
- [Stable neural networks and connections to continuous dynamical systems](https://mlsanity.com/paper/2510.22299/) (2025)

## Abstract (arXiv)

Despite the large success of deep neural networks (DNN) in recent years, most neural networks still lack mathematical guarantees in terms of stability. For instance, DNNs are vulnerable to small or even imperceptible input perturbations, so called adversarial examples, that can cause false predictions. This instability can have severe consequences in applications which influence the health and safety of humans, e.g., biomedical imaging or autonomous driving. While bounding the Lipschitz constant of a neural network improves stability, most methods rely on restricting the Lipschitz constants of each layer which gives a poor bound for the actual Lipschitz constant. In this paper we investigate a variational regularization method named CLIP for controlling the Lipschitz constant of a neural network, which can easily be integrated into the training procedure. We mathematically analyze the proposed model, in particular discussing the impact of the chosen regularization parameter on the output of the network. Finally, we numerically evaluate our method on both a nonlinear regression problem and the MNIST and Fashion-MNIST classification databases, and compare our results with a weight regularization approach.