mlsanityData to 7 Oct 2026

Some Fundamental Aspects about Lipschitz Continuity of Neural Networks

arXiv 2302.10886 ↗First posted 21 Feb 2023cs.LG stat.MLcited by 25

Not on the map. Why?

Abstract (arXiv)

Lipschitz continuity is a crucial functional property of any predictive model, that naturally governs its robustness, generalisation, as well as adversarial vulnerability. Contrary to other works that focus on obtaining tighter bounds and developing different practical strategies to enforce certain Lipschitz properties, we aim to thoroughly examine and characterise the Lipschitz behaviour of Neural Networks. Thus, we carry out an empirical investigation in a range of different settings (namely, architectures, datasets, label noise, and more) by exhausting the limits of the simplest and the most general lower and upper bounds. As a highlight of this investigation, we showcase a remarkable fidelity of the lower Lipschitz bound, identify a striking Double Descent trend in both upper and lower bounds to the Lipschitz and explain the intriguing effects of label noise on function smoothness and generalisation.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 58 of its 76 references that are in the map. First posted on arXiv 21 Feb 2023.

Builds on

58 of 76 references.

References · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  7. 2016Adversarial Machine Learning at Scalecited by 934
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  10. 2018On Lazy Training in Differentiable Programmingcited by 671
  11. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  12. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
38 more
  1. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  2. 2019Benign Overfitting in Linear Regressioncited by 544
  3. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  4. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  5. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  6. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  7. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  8. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 248
  9. 2018Sorting out Lipschitz function approximationcited by 211
  10. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  11. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  12. 2020The Lipschitz Constant of Self-Attentioncited by 171
  13. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 156
  14. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  15. 2021Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolationcited by 130
  16. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  17. 2021Globally-Robust Neural Networkscited by 106
  18. 2020Exactly Computing the Local Lipschitz Constant of ReLU Networkscited by 102
  19. 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
  20. 2020Training robust neural networks using Lipschitz boundscited by 98
  21. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  22. 2020Lipschitz Recurrent Neural Networkscited by 88
  23. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  24. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  25. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  26. 2021Skew Orthogonal Convolutionscited by 67
  27. 2017On the regularization of Wasserstein GANscited by 65
  28. 2018A Modern Take on the Bias-Variance Tradeoff in Neural Networkscited by 64
  29. 2021Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Boundscited by 64
  30. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  31. 2020Lipschitz Bounded Equilibrium Networkscited by 49
  32. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 49
  33. 2018Limitations of the Lipschitz constant as a defense against adversarial examplescited by 48
  34. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  35. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
  36. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  37. 2019Adversarial Lipschitz Regularizationcited by 29
  38. 2021CLIP: Cheap Lipschitz Training of Neural Networkscited by 29

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024Training Free Guided Flow Matching with Optimal Controlcited by 23
  2. 2025R-SCoRe: Revisiting Scene Coordinate Regression for Robust Large-Scale Visual Localizationcited by 16
  3. 2024Ferrari: Federated Feature Unlearning via Optimizing Feature Sensitivitycited by 11
  4. 2025Forget Forgetting: Continual Learning in a World of Abundant Memorycited by 9
  5. 2025RaanA: A Fast, Flexible, and Data-Efficient Post-Training Quantization Algorithmcited by 5
  6. 2024A Wasserstein perspective of Vanilla GANscited by 3
  7. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learningcited by 2
  8. 2024Making Classic GNNs Strong Baselines Across Varying Homophily: A Smoothness-Generalization Perspectivecited by 2
  9. 2026Learn to Evolve: Self-supervised Neural JKO Operator for Wasserstein Gradient Flowcited by 1
  10. 2026Generalizing GNNs with Tokenized Mixture of Expertscited by 1
  11. 2026LipsAM: Lipschitz-Continuous Amplitude Modifier for Audio Signal Processing and its Application to Plug-and-Play Dereverberationcited by 1
  12. 2026Fitting Is Not Enough: Smoothness in Extremely Quantized LLMscited by 1
13 more
  1. 2026AdvSerial: Physical Adversarial Attacks on Infrastructure-mounted Pedestrian Detectors via Semantic Feature Suppressioncited by 1
  2. 2025Learning Smooth State-Dependent Traversability from Dense Point Cloudscited by 0
  3. 2025Optimization-Induced Dynamics of Lipschitz Continuity in Neural Networkscited by 0
  4. 2025A Cycle-Consistency Constrained Framework for Dynamic Solution Space Reduction in Noninjective Regressioncited by 0
  5. 2025Spatial Lifting for Dense Predictioncited by 0
  6. 2025Scalable Equilibrium Propagation via Intermediate Error Signals for Deep Convolutional CRNNscited by 0
  7. 2025Noise-Guided Transport for Imitation Learningcited by 0
  8. 2025Delving into Cascaded Instability: A Lipschitz Continuity View on Image Restoration and Object Detection Synergycited by 0
  9. 2025Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activationscited by 0
  10. 2026The Price of Robustness: Stable Classifiers Need Overparameterizationcited by 0
  11. 2026Is Fairness Truly Fair? Towards Reliable Lipschitz Fairness in Multi-Task Learning via Fixed-texorpdfstringδdelta Alignmentcited by 0
  12. 2026Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Modelscited by 0
  13. 2026Spatial Lifting for Dense Predictioncited by 0