mlsanityData to 7 Oct 2026

Novel Quadratic Constraints for Extending LipSDP beyond Slope-Restricted Activations

arXiv 2401.14033 ↗First posted 25 Jan 2024cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recently, semidefinite programming (SDP) techniques have shown great promise in providing accurate Lipschitz bounds for neural networks. Specifically, the LipSDP approach (Fazlyab et al., 2019) has received much attention and provides the least conservative Lipschitz upper bounds that can be computed with polynomial time guarantees. However, one main restriction of LipSDP is that its formulation requires the activation functions to be slope-restricted on [0,1], preventing its further use for more general activation functions such as GroupSort, MaxMin, and Householder. One can rewrite MaxMin activations for example as residual ReLU networks. However, a direct application of LipSDP to the resultant residual ReLU networks is conservative and even fails in recovering the well-known fact that the MaxMin activation is 1-Lipschitz. Our paper bridges this gap and extends LipSDP beyond slope-restricted activation functions. To this end, we provide novel quadratic constraints for GroupSort, MaxMin, and Householder activations via leveraging their underlying properties such as sum preservation. Our proposed analysis is general and provides a unified approach for estimating ℓ2 and ℓ_∞ Lipschitz bounds for a rich class of neural network architectures, including non-residual and residual neural networks and implicit models, with GroupSort, MaxMin, and Householder activations. Finally, we illustrate the utility of our approach with a variety of experiments and show that our proposed SDPs generate less conservative Lipschitz bounds in comparison to existing approaches.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 47 of its 66 references that are in the map. First posted on arXiv 25 Jan 2024.

Builds on

47 of 66 references.

References · 47

  1. 2013Intriguing properties of neural networkscited by 5,114
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  4. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  5. 2019Deep Equilibrium Modelscited by 603
  6. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  7. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  8. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  9. 2019Provably Robust Deep Learning via Adversarially Trained Smoothed Classifierscited by 328
  10. 2018Sorting out Lipschitz function approximationcited by 211
  11. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  12. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
35 more
  1. 2021Globally-Robust Neural Networkscited by 106
  2. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  3. 2020Exactly Computing the Local Lipschitz Constant of ReLU Networkscited by 102
  4. 2020Training robust neural networks using Lipschitz boundscited by 98
  5. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  6. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  7. 2018Neural Lander: Stable Drone Landing Control using Learned Dynamicscited by 82
  8. 2018Generalizable Adversarial Training via Spectral Normalizationcited by 68
  9. 2021Skew Orthogonal Convolutionscited by 67
  10. 2021Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Boundscited by 64
  11. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  12. 2023A Unified Algebraic Perspective on Lipschitz Neural Networkscited by 58
  13. 2020Lipschitz Bounded Equilibrium Networkscited by 49
  14. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 49
  15. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  16. 2022Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagationcited by 45
  17. 2020Stability Analysis using Quadratic Constraints for Systems with Neural Network Controllerscited by 42
  18. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  19. 2022LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustnesscited by 32
  20. 2018Stability-certified reinforcement learning: A control-theoretic perspectivecited by 29
  21. 2020Approximating Lipschitz continuous functions with GroupSort neural networkscited by 27
  22. 2020Certifying Incremental Quadratic Constraints for Neural Networks via Convex Optimizationcited by 17
  23. 2022A Quantitative Geometric Approach to Neural-Network Smoothnesscited by 16
  24. 2022Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networkscited by 15
  25. 2022Lipschitz constant estimation for 1D convolutional neural networkscited by 13
  26. 2023Unlocking Deterministic Robustness Certification on ImageNetcited by 13
  27. 2019Universal Lipschitz Approximation in Bounded Depth Neural Networkscited by 11
  28. 2023Lipschitz-bounded 1D convolutional neural networks using the Cayley transform and the controllability Gramiancited by 9
  29. 2022Sparse Polynomial Optimisation for Neural Network Verificationcited by 8
  30. 2023Robust Machine Learning Modeling for Predictive Control Using Lipschitz-Constrained Neural Networkscited by 8
  31. 2023Convolutional Neural Networks as 2-D systemscited by 5
  32. 2023A Recipe for Improved Certifiable Robustnesscited by 4
  33. 2023A structure exploiting SDP solver for robust controller synthesiscited by 3
  34. 2023Certified Robust Models with Slack Control and Large Lipschitz Constantscited by 3
  35. 2015An Overview of Integral Quadratic Constraints for Delayed Nonlinear and Parameter-Varying Systemscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 1; 2026: 2.

6 papers.

  1. 2024Monotone, Bi-Lipschitz, and Polyak-Lojasiewicz Networkscited by 9
  2. 2024Lipschitz constant estimation for general neural network architectures using control toolscited by 8
  3. 2024A Complete Set of Quadratic Constraints for Repeated ReLU and Generalizationscited by 2
  4. 2025Improved Scalable Lipschitz Bounds for Deep Neural Networkscited by 2
  5. 2026ExLipBaB: Exact Lipschitz Constant Computation for Piecewise Linear Neural Networkscited by 1
  6. 2026Scalable Incremental Robustness Analysis of Neural Network Feedback Systemscited by 0