# Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant

arXiv: https://arxiv.org/abs/2001.06263 · HTML: https://mlsanity.com/paper/2001.06263/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2020.

## Built on by (most cited first, 25 of 25)

- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [CLIP: Cheap Lipschitz Training of Neural Networks](https://mlsanity.com/paper/2103.12531/) (2021)
- [Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions](https://mlsanity.com/paper/2210.16222/) (2022)
- [Augmented Shortcuts for Vision Transformers](https://mlsanity.com/paper/2106.15941/) (2021)
- [A Quantitative Geometric Approach to Neural-Network Smoothness](https://mlsanity.com/paper/2203.01212/) (2022)
- [Approximation of Lipschitz Functions using Deep Spline Neural Networks](https://mlsanity.com/paper/2204.06233/) (2022)
- [Provable Lipschitz Certification for Generative Models](https://mlsanity.com/paper/2107.02732/) (2021)
- [Analyzing Explainer Robustness via Probabilistic Lipschitzness of Prediction Functions](https://mlsanity.com/paper/2206.12481/) (2022)
- [Measuring Complexity of Learning Schemes Using Hessian-Schatten Total Variation](https://mlsanity.com/paper/2112.06209/) (2021)
- [Learning Gradually Non-convex Image Priors Using Score Matching](https://mlsanity.com/paper/2302.10502/) (2023)
- [Deep Stochastic Mechanics](https://mlsanity.com/paper/2305.19685/) (2023)
- [Sparsest Univariate Learning Models Under Lipschitz Constraint](https://mlsanity.com/paper/2112.13542/) (2021)
- [Uncertainty Estimation and Out-of-Distribution Detection for Deep Learning-Based Image Reconstruction using the Local Lipschitz](https://mlsanity.com/paper/2305.07618/) (2023)
- [A Tunable Despeckling Neural Network Stabilized via Diffusion Equation](https://mlsanity.com/paper/2411.15921/) (2024)
- [Distance-Aware Error for Spline Networks: A Bottom-Up Approach to Uncertainty](https://mlsanity.com/paper/2501.04757/) (2025)
- [1-Lipschitz Neural Networks are more expressive with N-Activations](https://mlsanity.com/paper/2311.06103/) (2023)
- [Learning Lipschitz Feedback Policies from Expert Demonstrations: Closed-Loop Guarantees, Generalization and Robustness](https://mlsanity.com/paper/2103.16629/) (2021)
- [CNN-based regularisation for CT image reconstructions](https://mlsanity.com/paper/2201.09132/) (2022)
- [L-Lipschitz Gershgorin ResNet Network](https://mlsanity.com/paper/2502.21279/) (2025)
- [Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks](https://mlsanity.com/paper/2509.18161/) (2025)
- [Worst-Case Distance-Aware Error Bounds for Neural Networks](https://mlsanity.com/paper/2510.22021/) (2025)
- [LDLT L-Lipschitz Network: Generalized Deep End-To-End Lipschitz Network Construction](https://mlsanity.com/paper/2512.05915/) (2025)
- [Optimized Piecewise Affine Abstractions of Neural Networks with Learnable Activation Functions](https://mlsanity.com/paper/2602.06737/) (2026)

## Abstract (arXiv)

We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the actual Lipschitz constant of the input-output relation. To that end, we first establish a global bound for the Lipschitz constant of neural networks. Based on the obtained bound, we then formulate a variational problem for learning activation functions. Our variational problem is infinite-dimensional and is not computationally tractable. However, we prove that there always exists a solution that has continuous and piecewise-linear (linear-spline) activations. This reduces the original problem to a finite-dimensional minimization where an l1 penalty on the parameters of the activations favors the learning of sparse nonlinearities. We numerically compare our scheme with standard ReLU network and its variations, PReLU and LeakyReLU and we empirically demonstrate the practical aspects of our framework.