# Deep Learning using Rectified Linear Units (ReLU)

arXiv: https://arxiv.org/abs/1803.08375 · HTML: https://mlsanity.com/paper/1803.08375/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Mar 2018.

## Built on by (most cited first, 30 of 48)

- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Tackling Data Heterogeneity in Federated Learning with Class Prototypes](https://mlsanity.com/paper/2212.02758/) (2022)
- [ViTPose++: Vision Transformer for Generic Body Pose Estimation](https://mlsanity.com/paper/2212.04246/) (2022)
- [HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers](https://mlsanity.com/paper/2211.08110/) (2022)
- [ThinkSwitcher: When to Think Hard, When to Think Fast](https://mlsanity.com/paper/2505.14183/) (2025)
- [Dual-side Sparse Tensor Core](https://mlsanity.com/paper/2105.09564/) (2021)
- [Hybrid quantum neural network for drug response prediction](https://mlsanity.com/paper/2211.05777/) (2022)
- [AdaWM: Adaptive World Model based Planning for Autonomous Driving](https://mlsanity.com/paper/2501.13072/) (2025)
- [Boost Vision Transformer with GPU-Friendly Sparsity and Quantization](https://mlsanity.com/paper/2305.10727/) (2023)
- [Towards Robust Natural-Looking Mammography Lesion Synthesis on Ipsilateral Dual-Views Breast Cancer Analysis](https://mlsanity.com/paper/2309.03506/) (2023)
- [Towards Infinite-Long Prefix in Transformer](https://mlsanity.com/paper/2406.14036/) (2024)
- [Multi-site, Multi-domain Airway Tree Modeling (ATM'22): A Public Benchmark for Pulmonary Airway Segmentation](https://mlsanity.com/paper/2303.05745/) (2023)
- [Hyperspectral Image Classification With Contrastive Graph Convolutional Network](https://mlsanity.com/paper/2205.11237/) (2022)
- [DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Experts](https://mlsanity.com/paper/2506.09351/) (2025)
- [Multi-hypothesis 3D human pose estimation metrics favor miscalibrated distributions](https://mlsanity.com/paper/2210.11179/) (2022)
- [Zero-Shot Transfer Learning for Structural Health Monitoring using Generative Adversarial Networks and Spectral Mapping](https://mlsanity.com/paper/2212.04002/) (2022)
- [Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attention](https://mlsanity.com/paper/2410.11222/) (2024)
- [Deep Learning Model Security: Threats and Defenses](https://mlsanity.com/paper/2412.08969/) (2024)
- [To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models](https://mlsanity.com/paper/2210.03005/) (2022)
- [Introducing Spectral Attention for Long-Range Dependency in Time Series Forecasting](https://mlsanity.com/paper/2410.20772/) (2024)
- [TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Model](https://mlsanity.com/paper/2510.16449/) (2025)
- [CapillaryNet: An Automated System to Quantify Skin Capillary Density and Red Blood Cell Velocity from Handheld Vital Microscopy](https://mlsanity.com/paper/2104.11574/) (2021)
- [Self-Supervised Visual Representation Learning on Food Images](https://mlsanity.com/paper/2303.09046/) (2023)
- [NanoVLMs: How small can we go and still make coherent Vision Language Models?](https://mlsanity.com/paper/2502.07838/) (2025)
- [Beyond Static Personas: Situational Personality Steering for Large Language Models](https://mlsanity.com/paper/2604.13846/) (2026)
- [Contrastive Graph Convolutional Networks for Hardware Trojan Detection in Third Party IP Cores](https://mlsanity.com/paper/2203.02095/) (2022)
- [Automatic stabilization of finite-element simulations using neural networks and hierarchical matrices](https://mlsanity.com/paper/2212.12695/) (2022)
- [Towards Robust Graph Structural Learning Beyond Homophily via Preserving Neighbor Similarity](https://mlsanity.com/paper/2401.09754/) (2024)
- [Detection of subclinical atherosclerosis by image-based deep learning on chest x-ray](https://mlsanity.com/paper/2403.18756/) (2024)
- [VFM-UDA++: Improving Network Architectures and Data Strategies for Unsupervised Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2503.10685/) (2025)

## Abstract (arXiv)

The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tracing the mathematical lineage of piecewise linear functions from early biological models to their definitive integration into deep learning by Nair & Hinton (2010). Alongside this historical rectification, we present a comprehensive empirical comparison of the ReLU, Hyperbolic Tangent (Tanh), and Logistic (Sigmoid) activation functions across image classification, text classification, and image reconstruction tasks. To ensure statistical robustness, we evaluated these functions using 10 independent randomized trials and assessed significance using the non-parametric Kruskal-Wallis H test. The empirical data validates the theoretical limitations of saturating functions. Sigmoid failed to converge in deep convolutional vision tasks due to the vanishing gradient problem, thus yielding accuracies equivalent to random probability. Conversely, ReLU and Tanh exhibited stable convergence. ReLU achieved the highest mean accuracy and F1-score on image classification and text classification tasks, while Tanh yielded the highest peak signal to noise ratio in image reconstruction. Ultimately, this study confirms a statistically significant performance variance among activations, thus reaffirming the necessity of non-saturating functions in deep architectures, and restores proper historical attribution to prior literature.