# Learning Activation Functions: A new paradigm for understanding Neural Networks

arXiv: https://arxiv.org/abs/1906.09529 · HTML: https://mlsanity.com/paper/1906.09529/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2019.

## Built on by (most cited first, 30 of 34)

- [KAN: Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2404.19756/) (2024)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions](https://mlsanity.com/paper/2105.09513/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Adaptive Rational Activations to Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09407/) (2021)
- [Learning Specialized Activation Functions for Physics-informed Neural Networks](https://mlsanity.com/paper/2308.04073/) (2023)
- [Gaussian Process Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2407.18397/) (2024)
- [Efficient Activation Function Optimization through Surrogate Modeling](https://mlsanity.com/paper/2301.05785/) (2023)
- [KAN we improve on HEP classification tasks? Kolmogorov-Arnold Networks applied to an LHC physics example](https://mlsanity.com/paper/2408.02743/) (2024)
- [K"ahler Geometry of Quiver Varieties and Machine Learning](https://mlsanity.com/paper/2101.11487/) (2021)
- [APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks](https://mlsanity.com/paper/2402.08244/) (2024)
- [More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations](https://mlsanity.com/paper/2605.26647/) (2026)
- [Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks](https://mlsanity.com/paper/2607.17624/) (2026)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [Efficient Search for Customized Activation Functions with Gradient Descent](https://mlsanity.com/paper/2408.06820/) (2024)
- [Learning Expressive Random Feature Models via Parametrized Activations](https://mlsanity.com/paper/2411.19468/) (2024)
- [Towards Identifiability of Interventional Stochastic Differential Equations](https://mlsanity.com/paper/2505.15987/) (2025)
- [Hybrid Ensemble Approaches: Optimal Deep Feature Fusion and Hyperparameter-Tuned Classifier Ensembling for Enhanced Brain Tumor Classification](https://mlsanity.com/paper/2507.12177/) (2025)
- [On the Generalization Properties of Learning the Random Feature Models with Learnable Activation Functions](https://mlsanity.com/paper/2510.15327/) (2025)
- [Tensor-based framework for training flexible neural networks](https://mlsanity.com/paper/2106.13542/) (2021)
- [Optimizing Neural Networks through Activation Function Discovery and Automatic Weight Initialization](https://mlsanity.com/paper/2304.03374/) (2023)
- [Activations Through Extensions: A Framework To Boost Performance Of Neural Networks](https://mlsanity.com/paper/2408.03599/) (2024)
- [SL2A-INR: Single-Layer Learnable Activation for Implicit Neural Representation](https://mlsanity.com/paper/2409.10836/) (2024)
- [On the Role of Activation Functions in EEG-To-Text Decoder](https://mlsanity.com/paper/2410.12572/) (2024)
- [Investigating Plausibility of Biologically Inspired Bayesian Learning in ANNs](https://mlsanity.com/paper/2411.18788/) (2024)
- [Polynomial, trigonometric, and tropical activations](https://mlsanity.com/paper/2502.01247/) (2025)
- [Tangma: A Tanh-Guided Activation Function with Learnable Parameters](https://mlsanity.com/paper/2507.10560/) (2025)
- [A parametric activation function based on Wendland RBF](https://mlsanity.com/paper/2507.11493/) (2025)
- [Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks](https://mlsanity.com/paper/2509.18161/) (2025)
- [Deformable Medical Image Registration with KAN-based Implicit Neural Representations](https://mlsanity.com/paper/2509.22874/) (2025)

## Abstract (arXiv)

The scope of research in the domain of activation functions remains limited and centered around improving the ease of optimization or generalization quality of neural networks (NNs). However, to develop a deeper understanding of deep learning, it becomes important to look at the non linear component of NNs more carefully. In this paper, we aim to provide a generic form of activation function along with appropriate mathematical grounding so as to allow for insights into the working of NNs in future. We propose "Self-Learnable Activation Functions" (SLAF), which are learned during training and are capable of approximating most of the existing activation functions. SLAF is given as a weighted sum of pre-defined basis elements which can serve for a good approximation of the optimal activation function. The coefficients for these basis elements allow a search in the entire space of continuous functions (consisting of all the conventional activations). We propose various training routines which can be used to achieve performance with SLAF equipped neural networks (SLNNs). We prove that SLNNs can approximate any neural network with lipschitz continuous activations, to any arbitrary error highlighting their capacity and possible equivalence with standard NNs. Also, SLNNs can be completely represented as a collections of finite degree polynomial upto the very last layer obviating several hyper parameters like width and depth. Since the optimization of SLNNs is still a challenge, we show that using SLAF along with standard activations (like ReLU) can provide performance improvements with only a small increase in number of parameters.