# Learning Activation Functions to Improve Deep Neural Networks

arXiv: https://arxiv.org/abs/1412.6830 · HTML: https://mlsanity.com/paper/1412.6830/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2014.

## Built on by (most cited first, 30 of 63)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples](https://mlsanity.com/paper/2209.02128/) (2022)
- [Deep Kronecker neural networks: A general framework for neural networks with adaptive activation functions](https://mlsanity.com/paper/2105.09513/) (2021)
- [When Deep Learning Meets Polyhedral Theory: A Survey](https://mlsanity.com/paper/2305.00241/) (2023)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions](https://mlsanity.com/paper/2210.16222/) (2022)
- [On the Number of Regions of Piecewise Linear Neural Networks](https://mlsanity.com/paper/2206.08615/) (2022)
- [Learning Specialized Activation Functions for Physics-informed Neural Networks](https://mlsanity.com/paper/2308.04073/) (2023)
- [Privacy and Trust Redefined in Federated Machine Learning](https://mlsanity.com/paper/2103.15753/) (2021)
- [Quantum Variational Activation Functions Empower Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2509.14026/) (2025)
- [Learning specialized activation functions with the Piecewise Linear Unit](https://mlsanity.com/paper/2104.03693/) (2021)
- [Piecewise Linear Neural Networks and Deep Learning](https://mlsanity.com/paper/2206.09149/) (2022)
- [Efficient Activation Function Optimization through Surrogate Modeling](https://mlsanity.com/paper/2301.05785/) (2023)
- [Kolmogorov-Arnold Networks in Low-Data Regimes: A Comparative Study with Multilayer Perceptrons](https://mlsanity.com/paper/2409.10463/) (2024)
- [Optimal Activation Functions for the Random Features Regression Model](https://mlsanity.com/paper/2206.01332/) (2022)
- [Neural Networks with A La Carte Selection of Activation Functions](https://mlsanity.com/paper/2206.12166/) (2022)
- [Increasing biases can be more efficient than increasing weights](https://mlsanity.com/paper/2301.00924/) (2023)
- [ENN: A Neural Network with DCT Adaptive Activation Functions](https://mlsanity.com/paper/2307.00673/) (2023)
- [Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2309.15244/) (2023)
- [KAN we improve on HEP classification tasks? Kolmogorov-Arnold Networks applied to an LHC physics example](https://mlsanity.com/paper/2408.02743/) (2024)
- [Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning](https://mlsanity.com/paper/2509.13240/) (2025)
- [Bayesian Neural Networks for Macroeconomic Analysis](https://mlsanity.com/paper/2211.04752/) (2022)
- [Optimal Nonlinearities Improve Generalization Performance of Random Features](https://mlsanity.com/paper/2309.16846/) (2023)
- [Adaptive Activation Functions for Predictive Modeling with Sparse Experimental Data](https://mlsanity.com/paper/2402.05401/) (2024)
- [More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations](https://mlsanity.com/paper/2605.26647/) (2026)
- [Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations](https://mlsanity.com/paper/2101.06848/) (2021)
- [Comparison of different convolutional neural network activation functions and methods for building ensembles](https://mlsanity.com/paper/2103.15898/) (2021)
- [Neural Network Verification as Piecewise Linear Optimization: Formulations for the Composition of Staircase Functions](https://mlsanity.com/paper/2211.14706/) (2022)
- [Empirical study of the modulus as activation function in computer vision applications](https://mlsanity.com/paper/2301.05993/) (2023)
- [From KAN to GR-KAN: Advancing Speech Enhancement with KAN-Based Methodology](https://mlsanity.com/paper/2412.17778/) (2024)

## Abstract (arXiv)

Artificial neural networks typically have a fixed, non-linear activation function at each neuron. We have designed a novel form of piecewise linear activation function that is learned independently for each neuron using gradient descent. With this adaptive activation function, we are able to improve upon deep neural network architectures composed of static rectified linear units, achieving state-of-the-art performance on CIFAR-10 (7.51%), CIFAR-100 (30.83%), and a benchmark from high-energy physics involving Higgs boson decay modes.