mlsanityData to 7 Oct 2026

Deep Learning using Rectified Linear Units (ReLU)

arXiv 1803.08375 ↗First posted 22 Mar 2018cs.NE cs.CV cs.LGcited by 43

Not on the map. Why?

Abstract (arXiv)

The Rectified Linear Unit (ReLU) is a foundational activation function in artficial neural networks. Recent literature frequently misattributes its origin to the 2018 (initial) version of this paper, which exclusively investigated ReLU at the classification layer. This paper formally corrects the citation record by tracing the mathematical lineage of piecewise linear functions from early biological models to their definitive integration into deep learning by Nair & Hinton (2010). Alongside this historical rectification, we present a comprehensive empirical comparison of the ReLU, Hyperbolic Tangent (Tanh), and Logistic (Sigmoid) activation functions across image classification, text classification, and image reconstruction tasks. To ensure statistical robustness, we evaluated these functions using 10 independent randomized trials and assessed significance using the non-parametric Kruskal-Wallis H test. The empirical data validates the theoretical limitations of saturating functions. Sigmoid failed to converge in deep convolutional vision tasks due to the vanishing gradient problem, thus yielding accuracies equivalent to random probability. Conversely, ReLU and Tanh exhibited stable convergence. ReLU achieved the highest mean accuracy and F1-score on image classification and text classification tasks, while Tanh yielded the highest peak signal to noise ratio in image reconstruction. Ultimately, this study confirms a statistically significant performance variance among activations, thus reaffirming the necessity of non-saturating functions in deep architectures, and restores proper historical attribution to prior literature.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 13; 2023: 5; 2024: 9; 2025: 8; 2026: 4.

43 papers.

Sort:

By area:

  1. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  2. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 66
  3. 2022ViTPose++: Vision Transformer for Generic Body Pose Estimationcited by 65
  4. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  5. 2025ThinkSwitcher: When to Think Hard, When to Think Fastcited by 37
  6. 2021Dual-side Sparse Tensor Corecited by 27
  7. 2022Hybrid quantum neural network for drug response predictioncited by 25
  8. 2025AdaWM: Adaptive World Model based Planning for Autonomous Drivingcited by 23
  9. 2023Boost Vision Transformer with GPU-Friendly Sparsity and Quantizationcited by 20
  10. 2023Towards Robust Natural-Looking Mammography Lesion Synthesis on Ipsilateral Dual-Views Breast Cancer Analysiscited by 10
  11. 2024Towards Infinite-Long Prefix in Transformercited by 10
  12. 2023Multi-site, Multi-domain Airway Tree Modeling (ATM'22): A Public Benchmark for Pulmonary Airway Segmentationcited by 9
31 more
  1. 2022Hyperspectral Image Classification With Contrastive Graph Convolutional Networkcited by 8
  2. 2025DIVE into MoE: Diversity-Enhanced Reconstruction of Large Language Models from Dense into Mixture-of-Expertscited by 7
  3. 2022Multi-hypothesis 3D human pose estimation metrics favor miscalibrated distributionscited by 6
  4. 2022Zero-Shot Transfer Learning for Structural Health Monitoring using Generative Adversarial Networks and Spectral Mappingcited by 6
  5. 2024Quadratic Gating Mixture of Experts: Statistical Insights into Self-Attentioncited by 4
  6. 2024Deep Learning Model Security: Threats and Defensescited by 4
  7. 2022To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Modelscited by 3
  8. 2024Introducing Spectral Attention for Long-Range Dependency in Time Series Forecastingcited by 3
  9. 2025TrajSelector: Harnessing Latent Representations for Efficient and Effective Best-of-N in Large Reasoning Modelcited by 3
  10. 2021CapillaryNet: An Automated System to Quantify Skin Capillary Density and Red Blood Cell Velocity from Handheld Vital Microscopycited by 2
  11. 2023Self-Supervised Visual Representation Learning on Food Imagescited by 2
  12. 2025NanoVLMs: How small can we go and still make coherent Vision Language Models?cited by 2
  13. 2026Beyond Static Personas: Situational Personality Steering for Large Language Modelscited by 2
  14. 2022Contrastive Graph Convolutional Networks for Hardware Trojan Detection in Third Party IP Corescited by 1
  15. 2022Automatic stabilization of finite-element simulations using neural networks and hierarchical matricescited by 1
  16. 2024Towards Robust Graph Structural Learning Beyond Homophily via Preserving Neighbor Similaritycited by 1
  17. 2024Detection of subclinical atherosclerosis by image-based deep learning on chest x-raycited by 1
  18. 2025VFM-UDA++: Improving Network Architectures and Data Strategies for Unsupervised Domain Adaptive Semantic Segmentationcited by 1
  19. 2021Transferring Voice Knowledge for Acoustic Event Detection: An Empirical Studycited by 0
  20. 2021EffCNet: An Efficient CondenseNet for Image Classification on NXP BlueBoxcited by 0
  21. 2022Accurate Portraits of Scientific Resources and Knowledge Service Componentscited by 0
  22. 2022CellTypeGraph: A New Geometric Computer Vision Benchmarkcited by 0
  23. 2023Dissecting Continual Learning a Structural and Data Analysiscited by 0
  24. 2024Radio U-Net: a convolutional neural network to detect diffuse radio sources in galaxy clusters and beyondcited by 0
  25. 2024A Visualized Malware Detection Framework with CNN and Conditional GANcited by 0
  26. 2024Self-supervised denoising of visual field data improves detection of glaucoma progressioncited by 0
  27. 2025COGITAO: A Visual Reasoning Framework To Study Compositionality & Generalizationcited by 0
  28. 2025Machine learning approach to single-shot multiparameter estimation for the non-linear Schr"odinger equationcited by 0
  29. 2026Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networkscited by 0
  30. 2026DaDaDa: A Dataset for Data Pricing in Data Marketplacescited by 0
  31. 2026S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modelingcited by 0