# A simple and efficient architecture for trainable activation functions

arXiv: https://arxiv.org/abs/1902.03306 · HTML: https://mlsanity.com/paper/1902.03306/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2019.

## Built on by (most cited first, 10 of 10)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild](https://mlsanity.com/paper/2503.10065/) (2025)
- [Curvature-based Feature Selection with Application in Classifying Electronic Health Records](https://mlsanity.com/paper/2101.03581/) (2021)
- [Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks](https://mlsanity.com/paper/2607.17624/) (2026)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [Towards Identifiability of Interventional Stochastic Differential Equations](https://mlsanity.com/paper/2505.15987/) (2025)
- [Activations Through Extensions: A Framework To Boost Performance Of Neural Networks](https://mlsanity.com/paper/2408.03599/) (2024)
- [Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metric](https://mlsanity.com/paper/2507.06765/) (2025)
- [Developing Training Procedures for Piecewise-linear Spline Activation Functions in Neural Networks](https://mlsanity.com/paper/2509.18161/) (2025)

## Abstract (arXiv)

Learning automatically the best activation function for the task is an active topic in neural network research. At the moment, despite promising results, it is still difficult to determine a method for learning an activation function that is at the same time theoretically simple and easy to implement. Moreover, most of the methods proposed so far introduce new parameters or adopt different learning techniques. In this work we propose a simple method to obtain trained activation function which adds to the neural network local subnetworks with a small amount of neurons. Experiments show that this approach could lead to better result with respect to using a pre-defined activation function, without introducing a large amount of extra parameters that need to be learned.