# Approximation of Lipschitz Functions using Deep Spline Neural Networks

arXiv: https://arxiv.org/abs/2204.06233 · HTML: https://mlsanity.com/paper/2204.06233/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 13 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Lipschitz regularity of deep neural networks: analysis and efficient estimation](https://mlsanity.com/paper/1805.10965/) (2018)
- [Regularisation of Neural Networks by Enforcing Lipschitz Continuity](https://mlsanity.com/paper/1804.04368/) (2018)
- [Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networks](https://mlsanity.com/paper/1906.04893/) (2019)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Plug-and-Play Methods Provably Converge with Properly Trained Denoisers](https://mlsanity.com/paper/1905.05406/) (2019)
- [Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networks](https://mlsanity.com/paper/1802.04034/) (2018)
- [Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems](https://mlsanity.com/paper/1704.03488/) (2017)
- [Training robust neural networks using Lipschitz bounds](https://mlsanity.com/paper/2005.02929/) (2020)
- [The Singular Values of Convolutional Layers](https://mlsanity.com/paper/1805.10408/) (2018)
- [Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks](https://mlsanity.com/paper/1709.06079/) (2017)
- [Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networks](https://mlsanity.com/paper/1911.00937/) (2019)
- [Lipschitz constant estimation of Neural Networks via sparse polynomial optimization](https://mlsanity.com/paper/2004.08688/) (2020)
- [ResNet with one-neuron hidden layers is a Universal Approximator](https://mlsanity.com/paper/1806.10909/) (2018)
- [Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolation](https://mlsanity.com/paper/1512.07331/) (2015)
- [Limitations of the Lipschitz constant as a defense against adversarial examples](https://mlsanity.com/paper/1807.09705/) (2018)
- [Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100](https://mlsanity.com/paper/2108.04062/) (2021)
- [A representer theorem for deep neural networks](https://mlsanity.com/paper/1802.09210/) (2018)
- [Convolutional Proximal Neural Networks and Plug-and-Play Algorithms](https://mlsanity.com/paper/2011.02281/) (2020)
- [Stabilizing Invertible Neural Networks Using Mixture Models](https://mlsanity.com/paper/2009.02994/) (2020)
- [CLIP: Cheap Lipschitz Training of Neural Networks](https://mlsanity.com/paper/2103.12531/) (2021)
- [Approximating Lipschitz continuous functions with GroupSort neural networks](https://mlsanity.com/paper/2006.05254/) (2020)
- [Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant](https://mlsanity.com/paper/2001.06263/) (2020)
- [Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)](https://mlsanity.com/paper/1604.02313/) (2016)
- [Higher-order total variation approaches and generalisations](https://mlsanity.com/paper/1912.01587/) (2019)
- [Universal Lipschitz Approximation in Bounded Depth Neural Networks](https://mlsanity.com/paper/1904.04861/) (2019)

## Built on by (most cited first, 15 of 15)

- [fKAN: Fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functions](https://mlsanity.com/paper/2406.07456/) (2024)
- [Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspective](https://mlsanity.com/paper/2210.01787/) (2022)
- [A Neural-Network-Based Convex Regularizer for Inverse Problems](https://mlsanity.com/paper/2211.12461/) (2022)
- [Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithms](https://mlsanity.com/paper/2308.10542/) (2023)
- [Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions](https://mlsanity.com/paper/2210.16222/) (2022)
- [On the Number of Regions of Piecewise Linear Neural Networks](https://mlsanity.com/paper/2206.08615/) (2022)
- [Provably Convergent Plug-and-Play Quasi-Newton Methods](https://mlsanity.com/paper/2303.07271/) (2023)
- [Approximation theory for 1-Lipschitz ResNets](https://mlsanity.com/paper/2505.12003/) (2025)
- [Recurrent Neural Networks and Universal Approximation of Bayesian Filters](https://mlsanity.com/paper/2211.00335/) (2022)
- [Approximation Theory for Lipschitz Continuous Transformers](https://mlsanity.com/paper/2602.15503/) (2026)
- [Universal Time-Uniform Trajectory Approximation for Random Dynamical Systems with Recurrent Neural Networks](https://mlsanity.com/paper/2211.08018/) (2022)
- [Compact: Approximating Complex Activation Functions for Secure Computation](https://mlsanity.com/paper/2309.04664/) (2023)
- [1-Lipschitz Neural Networks are more expressive with N-Activations](https://mlsanity.com/paper/2311.06103/) (2023)
- [Deep P-Spline: Theory, Fast Tuning, and Application](https://mlsanity.com/paper/2501.01376/) (2025)
- [Approximation of Maximally Monotone Operators : A Graph Convergence Perspective](https://mlsanity.com/paper/2605.12301/) (2026)

## Abstract (arXiv)

Lipschitz-constrained neural networks have many applications in machine learning. Since designing and training expressive Lipschitz-constrained networks is very challenging, there is a need for improved methods and a better theoretical understanding. Unfortunately, it turns out that ReLU networks have provable disadvantages in this setting. Hence, we propose to use learnable spline activation functions with at least 3 linear regions instead. We prove that this choice is optimal among all component-wise 1-Lipschitz activation functions in the sense that no other weight constrained architecture can approximate a larger class of functions. Additionally, this choice is at least as expressive as the recently introduced non component-wise Groupsort activation function for spectral-norm-constrained weights. Previously published numerical results support our theoretical findings.