mlsanityData to 7 Oct 2026

Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networks

arXiv 2104.05097 ↗First posted 11 Apr 2021cs.LG cs.AI stat.MLcited by 30

Not on the map. Why?

Abstract (arXiv)

Lipschitz constrained networks have gathered considerable attention in the deep learning community, with usages ranging from Wasserstein distance estimation to the training of certifiably robust classifiers. However they remain commonly considered as less accurate, and their properties in learning are still not fully understood. In this paper we clarify the matter: when it comes to classification 1-Lipschitz neural networks enjoy several advantages over their unconstrained counterpart. First, we show that these networks are as accurate as classical ones, and can fit arbitrarily difficult boundaries. Then, relying on a robustness metric that reflects operational needs we characterize the most robust classifier: the WGAN discriminator. Next, we show that 1-Lipschitz neural networks generalize well under milder assumptions. Finally, we show that hyper-parameters of the loss are crucial for controlling the accuracy-robustness trade-off. We conclude that they exhibit appealing properties to pave the way toward provably accurate, and provably robust neural networks.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 47 of its 90 references that are in the map. First posted on arXiv 11 Apr 2021.

Builds on

47 of 90 references.

References · 47

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2013Intriguing properties of neural networkscited by 5,114
  4. 2017Improved Training of Wasserstein GANscited by 2,223
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  6. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  7. 2018Robustness May Be at Odds with Accuracycited by 873
  8. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  9. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  10. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  11. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  12. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
35 more
  1. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  2. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  3. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  4. 2018Adversarial Attacks and Defences: A Surveycited by 259
  5. 2018Towards Fast Computation of Certified Robustness for ReLU Networkscited by 254
  6. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  7. 2018Sorting out Lipschitz function approximationcited by 211
  8. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  9. 2018Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approachcited by 181
  10. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  11. 2020The Lipschitz Constant of Self-Attentioncited by 171
  12. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  13. 2019Orthogonal Convolutional Neural Networkscited by 122
  14. 2016Robust Large Margin Deep Neural Networkscited by 117
  15. 2018Predicting the Generalization Gap in Deep Networks with Margin Distributionscited by 111
  16. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
  17. 2020Lipschitz Recurrent Neural Networkscited by 88
  18. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  19. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  20. 2017Orthogonal Recurrent Neural Networks with Scaled Cayley Transformcited by 82
  21. 2019Orthogonal Deep Neural Networkscited by 72
  22. 2016Efficient Orthogonal Parametrisation of Recurrent Neural Networks Using Householder Reflectionscited by 56
  23. 2018Limitations of the Lipschitz constant as a defense against adversarial examplescited by 48
  24. 2021Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neuronscited by 40
  25. 2021Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networkscited by 38
  26. 2021Fast and accurate optimization on the orthogonal manifold without retractioncited by 35
  27. 2017Towards Interpretable Deep Neural Networks by Leveraging Adversarial Examplescited by 27
  28. 2020Approximating Lipschitz continuous functions with GroupSort neural networkscited by 27
  29. 2020Achieving robustness in classification using optimal transport with hinge regularizationcited by 27
  30. 2019Cross-Entropy Loss and Low-Rank Features Have Responsibility for Adversarial Examplescited by 17
  31. 2016Learning Unitary Operators with Help From u(n)cited by 15
  32. 2021Convolutional Normalization: Improving Deep Convolutional Network Robustness and Trainingcited by 15
  33. 2013Efficient Classification for Metric Datacited by 9
  34. 2021Bridging the Gap Between Adversarial Robustness and Optimization Biascited by 5
  35. 2020Stochastic Flows and Geometric Optimization on the Orthogonal Groupcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 8; 2024: 6; 2025: 8; 2026: 6.

30 papers.

Sort:

By area:

  1. 2025Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retentioncited by 45
  2. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 25
  3. 2023Generalization bounds for neural ordinary differential equations and deep residual networkscited by 21
  4. 2025Training Transformers with Enforced Lipschitz Constantscited by 18
  5. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16
  6. 2024The Effect of Intrinsic Dataset Properties on Generalization: Unraveling Learning Differences Between Natural and Medical Imagescited by 13
  7. 2023Density-Softmax: Efficient Test-time Model for Uncertainty Estimation and Robustness under Distribution Shiftscited by 12
  8. 2023DP-SGD Without Clipping: The Lipschitz Neural Network Waycited by 11
  9. 2023Robust One-Class Classification with Signed Distance Function using 1-Lipschitz Neural Networkscited by 10
  10. 2025Pay Attention to Attention Distribution: A New Local Lipschitz Bound for Transformerscited by 10
  11. 2023Lipschitz-bounded 1D convolutional neural networks using the Cayley transform and the controllability Gramiancited by 9
  12. 2025An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architecturescited by 8
18 more
  1. 2021Robust and Provably Monotonic Networkscited by 6
  2. 2023The Lipschitz-Variance-Margin Tradeoff for Enhanced Randomized Smoothingcited by 6
  3. 2023Convolutional Neural Networks as 2-D systemscited by 5
  4. 2024Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoderscited by 5
  5. 2025Enhancing Certified Robustness via Block Reflector Orthogonal Layers and Logit Annealing Losscited by 5
  6. 2026Approximation Theory for Lipschitz Continuous Transformerscited by 5
  7. 2025Efficient Robust Conformal Prediction via Lipschitz-Bounded Networkscited by 4
  8. 2024Compositional Curvature Bounds for Deep Neural Networkscited by 3
  9. 2024Intriguing Properties of Robust Classificationcited by 3
  10. 2024Spectral regularization for adversarially-robust representation learningcited by 2
  11. 2024LaCoOT: Layer Collapse through Optimal Transportcited by 1
  12. 2025Image Coding for Machines via Feature-Preserving Rate-Distortion Optimizationcited by 1
  13. 2025Fast and Flexible Robustness Certificates for Semantic Segmentationcited by 1
  14. 2026Lipschitz Multiscale Deep Equilibrium Models: A Theoretically Guaranteed and Accelerated Approachcited by 1
  15. 2026Rethinking Intrinsic Dimension Estimation in Neural Representationscited by 0
  16. 2026Beyond Lipschitz: Data-Driven Robustness via Discrete Modulus of Continuitycited by 0
  17. 2026LipSSD: Lipschitz-Constrained Single-Shot Detection for Adversarially Robust Object Detectioncited by 0
  18. 2026LiST: Lipschitz Scaling Training for Robust and Calibrated Neural Networkscited by 0