mlsanityData to 7 Oct 2026

Approximation of Lipschitz Functions using Deep Spline Neural Networks

arXiv 2204.06233 ↗First posted 13 Apr 2022cs.LG math.OCcited by 15

Not on the map. Why?

Abstract (arXiv)

Lipschitz-constrained neural networks have many applications in machine learning. Since designing and training expressive Lipschitz-constrained networks is very challenging, there is a need for improved methods and a better theoretical understanding. Unfortunately, it turns out that ReLU networks have provable disadvantages in this setting. Hence, we propose to use learnable spline activation functions with at least 3 linear regions instead. We prove that this choice is optimal among all component-wise 1-Lipschitz activation functions in the sense that no other weight constrained architecture can approximate a larger class of functions. Additionally, this choice is at least as expressive as the recently introduced non component-wise Groupsort activation function for spectral-norm-constrained weights. Previously published numerical results support our theoretical findings.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 27 of its 43 references that are in the map. First posted on arXiv 13 Apr 2022.

Builds on

27 of 43 references.

References · 27

  1. 2017Improved Training of Wasserstein GANscited by 2,223
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  3. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  4. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  5. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  6. 2018Sorting out Lipschitz function approximationcited by 211
  7. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 197
  8. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 193
  9. 2017Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problemscited by 107
  10. 2020Training robust neural networks using Lipschitz boundscited by 98
  11. 2018The Singular Values of Convolutional Layerscited by 95
  12. 2017Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networkscited by 92
15 more
  1. 2020Lipschitz constant estimation of Neural Networks via sparse polynomial optimizationcited by 87
  2. 2019Preventing Gradient Attenuation in Lipschitz Constrained Convolutional Networkscited by 86
  3. 2015Plug-and-Play Priors for Bright Field Electron Tomography and Sparse Interpolationcited by 78
  4. 2018ResNet with one-neuron hidden layers is a Universal Approximatorcited by 78
  5. 2018Limitations of the Lipschitz constant as a defense against adversarial examplescited by 48
  6. 2021Improved deterministic l2 robustness on CIFAR-10 and CIFAR-100cited by 47
  7. 2018A representer theorem for deep neural networkscited by 38
  8. 2020Convolutional Proximal Neural Networks and Plug-and-Play Algorithmscited by 36
  9. 2020Stabilizing Invertible Neural Networks Using Mixture Modelscited by 30
  10. 2021CLIP: Cheap Lipschitz Training of Neural Networkscited by 29
  11. 2020Approximating Lipschitz continuous functions with GroupSort neural networkscited by 27
  12. 2020Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constantcited by 25
  13. 2016Norm-preserving Orthogonal Permutation Linear Unit Activation Functions (OPLU)cited by 15
  14. 2019Higher-order total variation approaches and generalisationscited by 14
  15. 2019Universal Lipschitz Approximation in Bounded Depth Neural Networkscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 4; 2024: 1; 2025: 2; 2026: 2.

15 papers.

Sort:

  1. 2024fKAN: Fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functionscited by 60
  2. 2022Rethinking Lipschitz Neural Networks and Certified Robustness: A Boolean Function Perspectivecited by 48
  3. 2022A Neural-Network-Based Convex Regularizer for Inverse Problemscited by 40
  4. 2023Learning Weakly Convex Regularizers for Convergent Image-Reconstruction Algorithmscited by 30
  5. 2022Improving Lipschitz-Constrained Neural Networks by Learning Activation Functionscited by 23
  6. 2022On the Number of Regions of Piecewise Linear Neural Networkscited by 21
  7. 2023Provably Convergent Plug-and-Play Quasi-Newton Methodscited by 15
  8. 2025Approximation theory for 1-Lipschitz ResNetscited by 7
  9. 2022Recurrent Neural Networks and Universal Approximation of Bayesian Filterscited by 5
  10. 2026Approximation Theory for Lipschitz Continuous Transformerscited by 5
  11. 2022Universal Time-Uniform Trajectory Approximation for Random Dynamical Systems with Recurrent Neural Networkscited by 1
  12. 2023Compact: Approximating Complex Activation Functions for Secure Computationcited by 1
3 more
  1. 20231-Lipschitz Neural Networks are more expressive with N-Activationscited by 1
  2. 2025Deep P-Spline: Theory, Fast Tuning, and Applicationcited by 1
  3. 2026Approximation of Maximally Monotone Operators : A Graph Convergence Perspectivecited by 0