# ALReLU: A different approach on Leaky ReLU activation function to improve Neural Networks Performance

arXiv: https://arxiv.org/abs/2012.07564 · HTML: https://mlsanity.com/paper/2012.07564/

## Key facts

- Cited by 7 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 11 Dec 2020.

## Built on by (most cited first, 7 of 7)

- [Homotopy Relaxation Training Algorithms for Infinite-Width Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/2309.15244/) (2023)
- [Rethinking Decoder Design: Improving Biomarker Segmentation Using Depth-to-Space Restoration and Residual Linear Attention](https://mlsanity.com/paper/2506.18335/) (2025)
- [Deep Multi-Task Learning for Malware Image Classification](https://mlsanity.com/paper/2405.05906/) (2024)
- [Estimating fire Duration using regression methods](https://mlsanity.com/paper/2308.08936/) (2023)
- [Parametric Leaky Tanh: A New Hybrid Activation Function for Deep Learning](https://mlsanity.com/paper/2310.07720/) (2023)
- [RepAct: The Re-parameterizable Adaptive Activation Function](https://mlsanity.com/paper/2407.00131/) (2024)
- [Pioneering Precision in Lumbar Spine MRI Segmentation with Advanced Deep Learning and Data Enhancement](https://mlsanity.com/paper/2409.06018/) (2024)

## Abstract (arXiv)

Despite the unresolved 'dying ReLU problem', the classical ReLU activation function (AF) has been extensively applied in Deep Neural Networks (DNN), in particular Convolutional Neural Networks (CNN), for image classification. The common gradient issues of ReLU pose challenges in applications on academy and industry sectors. Recent approaches for improvements are in a similar direction by just proposing variations of the AF, such as Leaky ReLU (LReLU), while maintaining the solution within the same unresolved gradient problems. In this paper, the Absolute Leaky ReLU (ALReLU) AF, a variation of LReLU, is proposed, as an alternative method to resolve the common 'dying ReLU problem' on NN-based algorithms for supervised learning. The experimental results demonstrate that by using the absolute values of LReLU's small negative gradient, has a significant improvement in comparison with LReLU and ReLU, on image classification of diseases such as COVID-19, text and tabular data classification tasks on five different datasets.