mlsanityData to 7 Oct 2026

1-Lipschitz Neural Networks are more expressive with N-Activations

arXiv 2311.06103 ↗First posted 10 Nov 2023cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

A crucial property for achieving secure, trustworthy and interpretable deep learning systems is their robustness: small changes to a system's inputs should not result in large changes to its outputs. Mathematically, this means one strives for networks with a small Lipschitz constant. Several recent works have focused on how to construct such Lipschitz networks, typically by imposing constraints on the weight matrices. In this work, we study an orthogonal aspect, namely the role of the activation function. We show that commonly used activation functions, such as MaxMin, as well as all piece-wise linear ones with two segments unnecessarily restrict the class of representable functions, even in the simplest one-dimensional setting. We furthermore introduce the new N-activation function that is provably more expressive than currently popular activation functions. We provide code at https://github.com/berndprach/NActivation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 24 of its 28 references that are in the map. First posted on arXiv 10 Nov 2023.

Builds on

24 of 28 references.

References · 24

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  4. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  5. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
  6. 2018Sorting out Lipschitz function approximationcited by 211
  7. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  8. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  9. 2021Globally-Robust Neural Networkscited by 106
  10. 2021Orthogonalizing Convolutional Layers with the Cayley Transformcited by 105
  11. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  12. 2021Skew Orthogonal Convolutionscited by 67
12 more
  1. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 59
  2. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  3. 2022Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networkscited by 39
  4. 2020Controllable Orthogonalization in Training DNNscited by 30
  5. 2020Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constantcited by 25
  6. 2022Improving Lipschitz-Constrained Neural Networks by Learning Activation Functionscited by 23
  7. 2022Approximation of Lipschitz Functions using Deep Spline Neural Networkscited by 15
  8. 2023Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernelscited by 13
  9. 2019Universal Lipschitz Approximation in Bounded Depth Neural Networkscited by 11
  10. 2020Lipschitz neural networks are dense in the set of all Lipschitz functionscited by 6
  11. 2022Robust-by-Design Classification via Unitary-Gradient Neural Networkscited by 4
  12. 2023Limitations of Piecewise Linearity for Efficient Robustness Certificationcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Intriguing Properties of Robust Classificationcited by 3