# Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

arXiv: https://arxiv.org/abs/2302.10886 · HTML: https://mlsanity.com/paper/2302.10886/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 76 references that are in the map.
- First posted on arXiv 21 Feb 2023.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [On Lazy Training in Differentiable Programming](https://mlsanity.com/paper/1812.07956/) (2018)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [Benign Overfitting in Linear Regression](https://mlsanity.com/paper/1906.11300/) (2019)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [The generalization error of random features regression: Precise asymptotics and double descent curve](https://mlsanity.com/paper/1908.05355/) (2019)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [The Marginal Value of Adaptive Gradient Methods in Machine Learning](https://mlsanity.com/paper/1705.08292/) (2017)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach](https://mlsanity.com/paper/1801.10578/) (2018)
- [The Lipschitz Constant of Self-Attention](https://mlsanity.com/paper/2006.04710/) (2020)
- [Finite Versus Infinite Neural Networks: an Empirical Study](https://mlsanity.com/paper/2007.15801/) (2020)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation](https://mlsanity.com/paper/2105.14368/) (2021)
- [Optimal Regularization Can Mitigate Double Descent](https://mlsanity.com/paper/2003.01897/) (2020)
- [Globally-Robust Neural Networks](https://mlsanity.com/paper/2102.08452/) (2021)
- [Exactly Computing the Local Lipschitz Constant of ReLU Networks](https://mlsanity.com/paper/2003.01219/) (2020)

## Built on by (most cited first, 26 of 26)

- [Training Free Guided Flow Matching with Optimal Control](https://mlsanity.com/paper/2410.18070/) (2024)
- [R-SCoRe: Revisiting Scene Coordinate Regression for Robust Large-Scale Visual Localization](https://mlsanity.com/paper/2501.01421/) (2025)
- [Ferrari: Federated Feature Unlearning via Optimizing Feature Sensitivity](https://mlsanity.com/paper/2405.17462/) (2024)
- [Forget Forgetting: Continual Learning in a World of Abundant Memory](https://mlsanity.com/paper/2502.07274/) (2025)
- [RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithm](https://mlsanity.com/paper/2504.03717/) (2025)
- [A Wasserstein perspective of Vanilla GANs](https://mlsanity.com/paper/2403.15312/) (2024)
- [Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2406.04088/) (2024)
- [Making Classic GNNs Strong Baselines Across Varying Homophily: A Smoothness-Generalization Perspective](https://mlsanity.com/paper/2412.09805/) (2024)
- [Optimization-Induced Dynamics of Lipschitz Continuity in Neural Networks](https://mlsanity.com/paper/2506.18588/) (2025)
- [Learn to Evolve: Self-supervised Neural JKO Operator for Wasserstein Gradient Flow](https://mlsanity.com/paper/2601.05583/) (2026)
- [Generalizing GNNs with Tokenized Mixture of Experts](https://mlsanity.com/paper/2602.09258/) (2026)
- [LipsAM: Lipschitz-Continuous Amplitude Modifier for Audio Signal Processing and its Application to Plug-and-Play Dereverberation](https://mlsanity.com/paper/2603.21684/) (2026)
- [Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs](https://mlsanity.com/paper/2605.08894/) (2026)
- [AdvSerial: Physical Adversarial Attacks on Infrastructure-mounted Pedestrian Detectors via Semantic Feature Suppression](https://mlsanity.com/paper/2607.17069/) (2026)
- [Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models](https://mlsanity.com/paper/2609.30935/) (2026)
- [Learning Smooth State-Dependent Traversability from Dense Point Clouds](https://mlsanity.com/paper/2506.04362/) (2025)
- [A Cycle-Consistency Constrained Framework for Dynamic Solution Space Reduction in Noninjective Regression](https://mlsanity.com/paper/2507.04659/) (2025)
- [Spatial Lifting for Dense Prediction](https://mlsanity.com/paper/2507.10222/) (2025)
- [Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNs](https://mlsanity.com/paper/2508.15989/) (2025)
- [Noise-Guided Transport for Imitation Learning](https://mlsanity.com/paper/2509.26294/) (2025)
- [Delving into Cascaded Instability: A Lipschitz Continuity View on Image Restoration and Object Detection Synergy](https://mlsanity.com/paper/2510.24232/) (2025)
- [Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations](https://mlsanity.com/paper/2510.24466/) (2025)
- [The Price of Robustness: Stable Classifiers Need Overparameterization](https://mlsanity.com/paper/2603.02806/) (2026)
- [Is Fairness Truly Fair? Towards Reliable Lipschitz Fairness in Multi-Task Learning via Fixed-texorpdfstringδdelta Alignment](https://mlsanity.com/paper/2606.10632/) (2026)
- [Spatial Lifting for Dense Prediction](https://mlsanity.com/paper/2610.00017/) (2026)
- [From Transformers to Weighted Automata: Towards the Verification of Large Language Models](https://mlsanity.com/paper/2610.04569/) (2026)

## Abstract (arXiv)

Lipschitz continuity is a crucial functional property of any predictive model, that naturally governs its robustness, generalisation, as well as adversarial vulnerability. Contrary to other works that focus on obtaining tighter bounds and developing different practical strategies to enforce certain Lipschitz properties, we aim to thoroughly examine and characterise the Lipschitz behaviour of Neural Networks. Thus, we carry out an empirical investigation in a range of different settings (namely, architectures, datasets, label noise, and more) by exhausting the limits of the simplest and the most general lower and upper bounds. As a highlight of this investigation, we showcase a remarkable fidelity of the lower Lipschitz bound, identify a striking Double Descent trend in both upper and lower bounds to the Lipschitz and explain the intriguing effects of label noise on function smoothness and generalisation.