# 1-Lipschitz Neural Networks are more expressive with N-Activations

arXiv: https://arxiv.org/abs/2311.06103 · HTML: https://mlsanity.com/paper/2311.06103/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 10 Nov 2023.

## Builds on (most cited first, 24 of 24)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Certified Adversarial Robustness via Randomized Smoothing](https://mlsanity.com/paper/1902.02918/) (2019)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [A Universal Law of Robustness via Isoperimetry](https://mlsanity.com/paper/2105.12806/) (2021)
- [Globally-Robust Neural Networks](https://mlsanity.com/paper/2102.08452/) (2021)
- [Orthogonalizing Convolutional Layers with the Cayley Transform](https://mlsanity.com/paper/2104.07167/) (2021)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Skew Orthogonal Convolutions](https://mlsanity.com/paper/2105.11417/) (2021)
- [A Dynamical System Perspective for Lipschitz Neural Networks](https://mlsanity.com/paper/2110.12690/) (2021)
- [Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100](https://mlsanity.com/paper/2108.04062/) (2021)
- [Almost-Orthogonal Layers for Efficient General-Purpose Lipschitz Networks](https://mlsanity.com/paper/2208.03160/) (2022)
- [Controllable Orthogonalization in Training DNNs](https://mlsanity.com/paper/2004.00917/) (2020)
- [Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant](https://mlsanity.com/paper/2001.06263/) (2020)
- [Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions](https://mlsanity.com/paper/2210.16222/) (2022)
- [Approximation of Lipschitz Functions using Deep Spline Neural Networks](https://mlsanity.com/paper/2204.06233/) (2022)
- [Beyond the Universal Law of Robustness: Sharper Laws for Random Features and Neural Tangent Kernels](https://mlsanity.com/paper/2302.01629/) (2023)
- [Universal Lipschitz Approximation in Bounded Depth Neural Networks](https://mlsanity.com/paper/1904.04861/) (2019)
- [Lipschitz neural networks are dense in the set of all Lipschitz functions](https://mlsanity.com/paper/2009.13881/) (2020)
- [Robust-by-Design Classification via Unitary-Gradient Neural Networks](https://mlsanity.com/paper/2209.04293/) (2022)
- [Limitations of Piecewise Linearity for Efficient Robustness Certification](https://mlsanity.com/paper/2301.08842/) (2023)

## Built on by (most cited first, 1 of 1)

- [Intriguing Properties of Robust Classification](https://mlsanity.com/paper/2412.04245/) (2024)

## Abstract (arXiv)

A crucial property for achieving secure, trustworthy and interpretable deep learning systems is their robustness: small changes to a system's inputs should not result in large changes to its outputs. Mathematically, this means one strives for networks with a small Lipschitz constant. Several recent works have focused on how to construct such Lipschitz networks, typically by imposing constraints on the weight matrices. In this work, we study an orthogonal aspect, namely the role of the activation function. We show that commonly used activation functions, such as MaxMin, as well as all piece-wise linear ones with two segments unnecessarily restrict the class of representable functions, even in the simplest one-dimensional setting. We furthermore introduce the new N-activation function that is provably more expressive than currently popular activation functions. We provide code at https://github.com/berndprach/NActivation.