# AReLU: Attention-based Rectified Linear Unit

arXiv: https://arxiv.org/abs/2006.13858 · HTML: https://mlsanity.com/paper/2006.13858/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2020.

## Built on by (most cited first, 9 of 9)

- [ToDo: Token Downsampling for Efficient Generation of High-Resolution Images](https://mlsanity.com/paper/2402.13573/) (2024)
- [DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing from Decentralised Data](https://mlsanity.com/paper/2104.11629/) (2021)
- [AASIST3: KAN-Enhanced AASIST Speech Deepfake Detection using SSL Features and Additional Regularization for the ASVspoof 2024 Challenge](https://mlsanity.com/paper/2408.17352/) (2024)
- [Don't Forget the Nonlinearity: Unlocking Activation Functions in Efficient Fine-Tuning](https://mlsanity.com/paper/2509.13240/) (2025)
- [Self-Adjust Softmax](https://mlsanity.com/paper/2502.18277/) (2025)
- [Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functions](https://mlsanity.com/paper/2402.09249/) (2024)
- [RepAct: The Re-parameterizable Adaptive Activation Function](https://mlsanity.com/paper/2407.00131/) (2024)
- [Interpretable Deep Regression Models with Interval-Censored Failure Time Data](https://mlsanity.com/paper/2503.19763/) (2025)
- [Automatic Contouring of Spinal Vertebrae on X-Ray using a Novel Sandwich U-Net Architecture](https://mlsanity.com/paper/2507.09158/) (2025)

## Abstract (arXiv)

Element-wise activation functions play a critical role in deep neural networks via affecting the expressivity power and the learning dynamics. Learning-based activation functions have recently gained increasing attention and success. We propose a new perspective of learnable activation function through formulating them with element-wise attention mechanism. In each network layer, we devise an attention module which learns an element-wise, sign-based attention map for the pre-activation feature map. The attention map scales an element based on its sign. Adding the attention module with a rectified linear unit (ReLU) results in an amplification of positive elements and a suppression of negative ones, both with learned, data-adaptive parameters. We coin the resulting activation function Attention-based Rectified Linear Unit (AReLU). The attention module essentially learns an element-wise residue of the activated part of the input, as ReLU can be viewed as an identity transformation. This makes the network training more resistant to gradient vanishing. The learned attentive activation leads to well-focused activation of relevant regions of a feature map. Through extensive evaluations, we show that AReLU significantly boosts the performance of most mainstream network architectures with only two extra learnable parameters per layer introduced. Notably, AReLU facilitates fast network training under small learning rates, which makes it especially suited in the case of transfer learning and meta learning. Our source code has been released (see https://github.com/densechen/AReLU).