mlsanityData to 14 Oct 2026

Sorting out Lipschitz function approximation

arXiv 1811.05381 ↗First posted 13 Nov 2018cs.LG stat.MLcited by 213

Not on the map. Why?

Abstract (arXiv)

Training neural networks under a strict Lipschitz constraint is useful for provable adversarial robustness, generalization bounds, interpretable gradients, and Wasserstein distance estimation. By the composition property of Lipschitz functions, it suffices to ensure that each individual affine transformation or nonlinear activation is 1-Lipschitz. The challenge is to do this while maintaining the expressive power. We identify a necessary property for such an architecture: each of the layers must preserve the gradient norm during backpropagation. Based on this, we propose to combine a gradient norm preserving activation function, GroupSort, with norm-constrained weight matrices. We show that norm-constrained GroupSort architectures are universal Lipschitz function approximators. Empirically, we show that norm-constrained GroupSort networks achieve tighter estimates of Wasserstein distance than their ReLU counterparts and can achieve provable adversarial robustness guarantees with little cost to accuracy.

Key facts. Cited by 213 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 49; 2022: 30; 2023: 41; 2024: 32; 2025: 33; 2026: 28.

213 papers.

Sort:

By area:

  1. 2021Learning to Optimize: A Primer and A Benchmarkcited by 208
  2. 2021Globally-Robust Neural Networkscited by 108
  3. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  4. 2021JFB: Jacobian-Free Backpropagation for Implicit Networkscited by 86
  5. 2022Learning Smooth Neural Functions via Lipschitz Regularizationcited by 86
  6. 2021Generative Modeling with Optimal Transport Mapscited by 82
  7. 2021Skew Orthogonal Convolutionscited by 67
  8. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  9. 2021Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Boundscited by 65
  10. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 60
  11. 2023A Unified Algebraic Perspective on Lipschitz Neural Networkscited by 60
  12. 2022Certified Training: Small Boxes are All You Needcited by 52
38 more
  1. 2022Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspectivecited by 51
  2. 2023Direct Parameterization of Lipschitz-Bounded Deep Networkscited by 51
  3. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  4. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  5. 2022Efficiently Computing Local Lipschitz Constants of Neural Networks via Bound Propagationcited by 45
  6. 2023How Smooth Is Attention?cited by 44
  7. 2022A Neural-Network-Based Convex Regularizer for Inverse Problemscited by 41
  8. 2021Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neuronscited by 40
  9. 2021Improved, Deterministic Smoothing for L_1 Certified Robustnesscited by 39
  10. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  11. 2022On the Effectiveness of Lipschitz-Driven Rehearsal in Continual Learningcited by 39
  12. 2023When Deep Learning Meets Polyhedral Theory: A Surveycited by 39
  13. 2021Wasserstein GANs Work Because They Fail (to Approximate the Wasserstein Distance)cited by 35
  14. 2021Neural Network Robustness as a Verification Property: A Principled Case Studycited by 35
  15. 2022LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustnesscited by 33
  16. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  17. 2022Universal approximation property of invertible neural networkscited by 30
  18. 2021Low Curvature Activations Reduce Overfitting in Adversarial Trainingcited by 29
  19. 2021CLIP: Cheap Lipschitz Training of Neural Networkscited by 29
  20. 2024LLM-Empowered State Representation for Reinforcement Learningcited by 28
  21. 2023Some Fundamental Aspects about Lipschitz Continuity of Neural Networkscited by 26
  22. 2024Distributional Preference Alignment of LLMs via Optimal Transportcited by 26
  23. 2021Intriguing Properties of Input-dependent Randomized Smoothingcited by 24
  24. 2021Boosting the Certified Robustness of L-infinity Distance Netscited by 24
  25. 2022Improving Lipschitz-Constrained Neural Networks by Learning Activation Functionscited by 24
  26. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  27. 2023Certified Robustness via Dynamic Margin Maximization and Improved Lipschitz Regularizationcited by 21
  28. 2023Riemannian Residual Neural Networkscited by 21
  29. 2024Parseval Regularization for Continual Reinforcement Learningcited by 21
  30. 2025Training Transformers with Enforced Lipschitz Constantscited by 20
  31. 2021CC-Cert: A Probabilistic Approach to Certify General Robustness of Neural Networkscited by 19
  32. 2022Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport?cited by 19
  33. 2023Efficient Bound of Lipschitz Constant for Convolutional Layers by Gram Iterationcited by 19
  34. 2021Invertible DenseNets with Concatenated LipSwishcited by 17
  35. 2021Input Convex Gradient Networkscited by 17
  36. 2025ENFORCE: Nonlinear Constrained Learning with Adaptive-depth Neural Projectioncited by 17
  37. 2021Learning Prediction Intervals for Regression: Generalization and Calibrationcited by 16
  38. 2021Existence, Stability and Scalability of Orthogonal Convolutional Neural Networkscited by 16

50 most cited of 213 All 213 (JSON)