mlsanityData to 7 Oct 2026

Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions

arXiv 2210.16222 ↗First posted 28 Oct 2022cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Lipschitz-constrained neural networks have several advantages over unconstrained ones and can be applied to a variety of problems, making them a topic of attention in the deep learning community. Unfortunately, it has been shown both theoretically and empirically that they perform poorly when equipped with ReLU activation functions. By contrast, neural networks with learnable 1-Lipschitz linear splines are known to be more expressive. In this paper, we show that such networks correspond to global optima of a constrained functional optimization problem that consists of the training of a neural network composed of 1-Lipschitz linear layers and 1-Lipschitz freeform activation functions with second-order total-variation regularization. Further, we propose an efficient method to train these neural networks. Our numerical experiments show that our trained networks compare favorably with existing 1-Lipschitz neural architectures.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 45 of its 68 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

45 of 68 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  5. 2017Improved Training of Wasserstein GANscited by 2,223
  6. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  7. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  8. 2018Robustness May Be at Odds with Accuracycited by 873
  9. 2016InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Netscited by 858
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  11. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  12. 2017Exploring Generalization in Deep Learningcited by 595
33 more
  1. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 464
  2. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  3. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  4. 2017Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing their Input Gradientscited by 236
  5. 2018Sorting out Lipschitz function approximationcited by 211
  6. 2019On instabilities of deep learning in image reconstruction - Does AI come at a cost?cited by 211
  7. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 197
  8. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  9. 2016Robust Large Margin Deep Neural Networkscited by 117
  10. 2017Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problemscited by 107
  11. 2021Gradient Step Denoiser for convergent Plug-and-Playcited by 107
  12. 2020Training robust neural networks using Lipschitz boundscited by 98
  13. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  14. 2015Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolationcited by 78
  15. 2014Learning Activation Functions to Improve Deep Neural Networkscited by 62
  16. 2020Learned convex regularizers for inverse problemscited by 54
  17. 2020Scalable Plug-and-Play ADMM with Convergence Guaranteescited by 49
  18. 2018Limitations of the Lipschitz constant as a defense against adversarial examplescited by 48
  19. 2022Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspectivecited by 48
  20. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  21. 2021Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neuronscited by 40
  22. 2018A representer theorem for deep neural networkscited by 38
  23. 2020Convolutional Proximal Neural Networks and Plug-and-Play Algorithmscited by 36
  24. 2017ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matchingcited by 32
  25. 2020Stabilizing Invertible Neural Networks Using Mixture Modelscited by 30
  26. 2021CLIP: Cheap Lipschitz Training of Neural Networkscited by 29
  27. 2015Deep Learning with S-shaped Rectified Linear Activation Unitscited by 26
  28. 2020Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constantcited by 25
  29. 2022Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport?cited by 19
  30. 2022Approximation of Lipschitz Functions using Deep Spline Neural Networkscited by 15
  31. 2020Sparsest Piecewise-Linear Regression of One-Dimensional Datacited by 12
  32. 2022Explicit representations for Banach subspaces of Lizorkin distributionscited by 3
  33. 2021Sparsest Univariate Learning Models Under Lipschitz Constraintcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 3; 2025: 7; 2026: 8.

23 papers.

Sort:

  1. 2022A Neural-Network-Based Convex Regularizer for Inverse Problemscited by 40
  2. 2023Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithmscited by 30
  3. 2025Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wildcited by 13
  4. 2025Learning Regularization Functionals for Inverse Problems: A Comparative Studycited by 12
  5. 2023Robustness of Deep Learning for Accelerated MRI: Benefits of Diverse Training Datacited by 10
  6. 2025DEALing with Image Reconstruction: Deep Attentive Least Squarescited by 6
  7. 2024Iteratively Refined Image Reconstruction with Learned Attentive Regularizerscited by 3
  8. 2024Do stable neural networks exist for classification problems? -- A new view on stability in AIcited by 2
  9. 2024Parseval Convolution Operators and Neural Networkscited by 2
  10. 2025Multivariate Fields of Experts for Convergent Image Reconstructioncited by 2
  11. 2026Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Taskscited by 2
  12. 20231-Lipschitz Neural Networks are more expressive with N-Activationscited by 1
11 more
  1. 2023Spatial Bayesian Neural Networkscited by 1
  2. 2026ExLipBaB: Exact Lipschitz Constant Computation for Piecewise Linear Neural Networkscited by 1
  3. 2026LipsAM: Lipschitz-Continuous Amplitude Modifier for Audio Signal Processing and its Application to Plug-and-Play Dereverberationcited by 1
  4. 2026Statistically Undetectable Backdoors in Deep Neural Networkscited by 1
  5. 2025Semialgebraic Neural Networks: From roots to representationscited by 0
  6. 2025Bridging GANs and Bayesian Neural Networks via Partial Stochasticitycited by 0
  7. 2025Topologically-Stabilized Graph Neural Networks: Empirical Robustness Across Domainscited by 0
  8. 2026Provably Contractive and High-Quality Denoisers for Convergent Restorationcited by 0
  9. 2026Emulating Non-Differentiable Metrics via Knowledge-Guided Learning: Introducing the Minkowski Image Losscited by 0
  10. 2026Optimizing Server Placement for Vertical Federated Learning in Dynamic Edge/Fog Networkscited by 0
  11. 2026Trainable Nonexpansive Denoisers for Contractive Image Reconstructioncited by 0