mlsanityData to 14 Oct 2026

Lipschitz regularity of deep neural networks: analysis and efficient estimation

arXiv 1805.10965 ↗First posted 28 May 2018stat.ML cs.LGcited by 415

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are notorious for being sensitive to small well-chosen perturbations, and estimating the regularity of such architectures is of utmost importance for safe and robust practical applications. In this paper, we investigate one of the key characteristics to assess the regularity of such methods: the Lipschitz constant of deep learning architectures. First, we show that, even for two layer neural networks, the exact computation of this quantity is NP-hard and state-of-art methods may significantly overestimate it. Then, we both extend and improve previous estimation methods by providing AutoLip, the first generic algorithm for upper bounding the Lipschitz constant of any automatically differentiable function. We provide a power method algorithm working with automatic differentiation, allowing efficient computations even on large convolutions. Second, for sequential neural networks, we propose an improved algorithm named SeqLip that takes advantage of the linear computation graph to split the computation per pair of consecutive layers. Third we propose heuristics on SeqLip in order to tackle very large networks. Our experiments show that SeqLip can significantly improve on the existing upper bounds. Finally, we provide an implementation of AutoLip in the PyTorch environment that may be used to better estimate the robustness of a given neural network to small perturbations or regularize it using more precise Lipschitz estimations.

Key facts. Cited by 415 papers in the map (data through 14 Oct 2026). First posted on arXiv 28 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 63; 2022: 65; 2023: 63; 2024: 82; 2025: 64; 2026: 78.

415 papers.

Sort:

By area:

  1. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
  2. 2021Machine Unlearning of Features and Labelscited by 229
  3. 2022Vision GNN: An Image is Worth Graph of Nodescited by 186
  4. 2021Is Homophily a Necessity for Graph Neural Networks?cited by 182
  5. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Datacited by 174
  6. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 135
  7. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  8. 2022Revisiting Over-smoothing and Over-squashing Using Ollivier-Ricci Curvaturecited by 98
  9. 2021Proximal Optimal Transport Modeling of Population Dynamicscited by 93
  10. 2022Learning Smooth Neural Functions via Lipschitz Regularizationcited by 86
  11. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 81
  12. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
38 more
  1. 2021Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Boundscited by 65
  2. 2022Contrastive Adapters for Foundation Model Group Robustnesscited by 62
  3. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 60
  4. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  5. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 51
  6. 2024CViT: Continuous Vision Transformer for Operator Learningcited by 51
  7. 2021Semi-Supervised Active Learning with Temporal Output Discrepancycited by 50
  8. 2022Score-based Generative Modeling Secretly Minimizes the Wasserstein Distancecited by 50
  9. 2021Recurrent Equilibrium Networks: Flexible Dynamic Models with Guaranteed Stability and Robustnesscited by 45
  10. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  11. 2022Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagationcited by 45
  12. 2023RPCANet: Deep Unfolding RPCA Based Infrared Small Target Detectioncited by 44
  13. 2023How Smooth Is Attention?cited by 44
  14. 2024INViT: A Generalizable Routing Problem Solver with Invariant Nested View Transformercited by 44
  15. 2021Solving PDE-constrained Control Problems Using Operator Learningcited by 42
  16. 2023NECO: NEural Collapse Based Out-of-distribution detectioncited by 42
  17. 2022Can Push-forward Generative Models Fit Multimodal Distributions?cited by 41
  18. 2022Tree Mover's Distance: Bridging Graph Metrics and Stability of Graph Neural Networkscited by 41
  19. 2023Towards Understanding the Generalization of Graph Neural Networkscited by 41
  20. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 39
  21. 2021Understanding the role of importance weighting for deep learningcited by 39
  22. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  23. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  24. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  25. 2023Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discoverycited by 39
  26. 2021Learning Frequency Domain Approximation for Binary Neural Networkscited by 38
  27. 2024Stable Neural Stochastic Differential Equations in Analyzing Irregular Time Series Datacited by 36
  28. 2024Visual Adversarial Attack on Vision-Language Models for Autonomous Drivingcited by 36
  29. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 34
  30. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  31. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  32. 2024Continuous Ensemble Weather Forecasting with Diffusion modelscited by 30
  33. 2021CLIP: Cheap Lipschitz Training of Neural Networkscited by 29
  34. 2023Physics-Informed Machine Learning for Modeling and Control of Dynamical Systemscited by 28
  35. 2022L2C2: Locally Lipschitz Continuous Constraint towards Stable and Smooth Reinforcement Learningcited by 27
  36. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 26
  37. 2021Measuring Generalization with Optimal Transportcited by 25
  38. 2021Gradient Normalization for Generative Adversarial Networkscited by 25

50 most cited of 415 All 415 (JSON)