# Learning activation functions from data using cubic spline interpolation

arXiv: https://arxiv.org/abs/1605.05509 · HTML: https://mlsanity.com/paper/1605.05509/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2016.

## Built on by (most cited first, 13 of 13)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [fKAN: Fractional Kolmogorov-Arnold Networks with trainable Jacobi basis functions](https://mlsanity.com/paper/2406.07456/) (2024)
- [ExSpliNet: An interpretable and expressive spline-based neural network](https://mlsanity.com/paper/2205.01510/) (2022)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [A methodology for training homomorphicencryption friendly neural networks](https://mlsanity.com/paper/2111.03362/) (2021)
- [KASAM: Spline Additive Models for Function Approximation](https://mlsanity.com/paper/2205.06376/) (2022)
- [The limitation of neural nets for approximation and optimization](https://mlsanity.com/paper/2311.12253/) (2023)
- [Distal Interference: Exploring the Limits of Model-Based Continual Learning](https://mlsanity.com/paper/2402.08255/) (2024)
- [Introducing the DOME Activation Functions](https://mlsanity.com/paper/2109.14798/) (2021)
- [ATLAS: Universal Function Approximator for Memory Retention](https://mlsanity.com/paper/2208.05388/) (2022)
- [Bayesian optimization for sparse neural networks with trainable activation functions](https://mlsanity.com/paper/2304.04455/) (2023)
- [Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks](https://mlsanity.com/paper/2509.18161/) (2025)
- [TESLA: Taylor Expansion of Sinusoidal Learnable Activations](https://mlsanity.com/paper/2608.11970/) (2026)

## Abstract (arXiv)

Neural networks require a careful design in order to perform properly on a given task. In particular, selecting a good activation function (possibly in a data-dependent fashion) is a crucial step, which remains an open problem in the research community. Despite a large amount of investigations, most current implementations simply select one fixed function from a small set of candidates, which is not adapted during training, and is shared among all neurons throughout the different layers. However, neither two of these assumptions can be supposed optimal in practice. In this paper, we present a principled way to have data-dependent adaptation of the activation functions, which is performed independently for each neuron. This is achieved by leveraging over past and present advances on cubic spline interpolation, allowing for local adaptation of the functions around their regions of use. The resulting algorithm is relatively cheap to implement, and overfitting is counterbalanced by the inclusion of a novel damping criterion, which penalizes unwanted oscillations from a predefined shape. Experimental results validate the proposal over two well-known benchmarks.