# Nonlinearity Enhanced Adaptive Activation Functions

arXiv: https://arxiv.org/abs/2403.19896 · HTML: https://mlsanity.com/paper/2403.19896/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 36 references that are in the map.
- First posted on arXiv 29 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [A high-bias, low-variance introduction to Machine Learning for physicists](https://mlsanity.com/paper/1803.08823/) (2018)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [A survey on modern trainable activation functions](https://mlsanity.com/paper/2005.00817/) (2020)
- [Learning Activation Functions to Improve Deep Neural Networks](https://mlsanity.com/paper/1412.6830/) (2014)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Learning Combinations of Activation Functions](https://mlsanity.com/paper/1801.09403/) (2018)
- [Adaptive Blending Units: Trainable Activation Functions for Deep Neural Networks](https://mlsanity.com/paper/1806.10064/) (2018)
- [Activation Functions in Artificial Neural Networks: A Systematic Overview](https://mlsanity.com/paper/2101.09957/) (2021)
- [E-swish: Adjusting Activations to Different Network Depths](https://mlsanity.com/paper/1801.07145/) (2018)
- [A survey on recently proposed activation functions for Deep Learning](https://mlsanity.com/paper/2204.02921/) (2022)
- [MaxGain: Regularisation of Neural Networks by Constraining Activation Magnitudes](https://mlsanity.com/paper/1804.05965/) (2018)
- [Formalising the Use of the Activation Function in Neural Inference](https://mlsanity.com/paper/2102.04896/) (2021)
- [Neural Network Characterization and Entropy Regulated Data Balancing through Principal Component Analysis](https://mlsanity.com/paper/2312.01392/) (2023)

## Abstract (arXiv)

A general procedure for introducing parametric, learned, nonlinearity into activation functions is found to enhance the accuracy of representative neural networks without requiring significant additional computational resources. Examples are given based on the standard rectified linear unit (ReLU) as well as several other frequently employed activation functions. The associated accuracy improvement is quantified both in the context of the MNIST digit data set and a convolutional neural network (CNN) benchmark example.