# Zorro: A Flexible and Differentiable Parametric Family of Activation Functions That Extends ReLU and GELU

arXiv: https://arxiv.org/abs/2409.19239 · HTML: https://mlsanity.com/paper/2409.19239/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 32 references that are in the map.
- First posted on arXiv 28 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Vision Transformer for Small-Size Datasets](https://mlsanity.com/paper/2112.13492/) (2021)
- [Adaptive Rational Activations to Boost Deep Reinforcement Learning](https://mlsanity.com/paper/2102.09407/) (2021)
- [Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks](https://mlsanity.com/paper/2402.09092/) (2024)
- [GELU Activation Function in Deep Learning: A Comprehensive Mathematical Analysis and Performance](https://mlsanity.com/paper/2305.12073/) (2023)
- [HOSC: A Periodic Activation Function for Preserving Sharp Features in Implicit Neural Representations](https://mlsanity.com/paper/2401.10967/) (2024)
- [ENN: A Neural Network with DCT Adaptive Activation Functions](https://mlsanity.com/paper/2307.00673/) (2023)
- [A Method on Searching Better Activation Functions](https://mlsanity.com/paper/2405.12954/) (2024)
- [Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural Networks to Recover Nonlinear Oscillations Efficiently](https://mlsanity.com/paper/2304.09759/) (2023)
- [ErfReLU: Adaptive Activation Function for Deep Neural Network](https://mlsanity.com/paper/2306.01822/) (2023)
- [A Non-monotonic Smooth Activation Function](https://mlsanity.com/paper/2310.10126/) (2023)
- [APALU: A Trainable, Adaptive Activation Function for Deep Learning Networks](https://mlsanity.com/paper/2402.08244/) (2024)
- [Parametric Leaky Tanh: A New Hybrid Activation Function for Deep Learning](https://mlsanity.com/paper/2310.07720/) (2023)

## Abstract (arXiv)

Even in recent neural network architectures such as Transformers and Extended LSTM (xLSTM), and traditional ones like Convolutional Neural Networks, Activation Functions are an integral part of nearly all neural networks. They enable more effective training and capture nonlinear data patterns. More than 400 functions have been proposed over the last 30 years, including fixed or trainable parameters, but only a few are widely used. ReLU is one of the most frequently used, with GELU and Swish variants increasingly appearing. However, ReLU presents non-differentiable points and exploding gradient issues, while testing different parameters of GELU and Swish variants produces varying results, needing more parameters to adapt to datasets and architectures. This article introduces a novel set of activation functions called Zorro, a continuously differentiable and flexible family comprising five main functions fusing ReLU and Sigmoid. Zorro functions are smooth and adaptable, and serve as information gates, aligning with ReLU in the 0-1 range, offering an alternative to ReLU without the need for normalization, neuron death, or gradient explosions. Zorro also approximates functions like Swish, GELU, and DGELU, providing parameters to adjust to different datasets and architectures. We tested it on fully connected, convolutional, and transformer architectures to demonstrate its effectiveness.