mlsanityData to 14 Oct 2026

Empirical Evaluation of Rectified Activations in Convolutional Network

arXiv 1505.00853 ↗First posted 5 May 2015cs.LG cs.CV stat.MLcited by 519

Not on the map. Why?

Abstract (arXiv)

In this paper we investigate the performance of different types of rectified activation functions in convolutional neural network: standard rectified linear unit (ReLU), leaky rectified linear unit (Leaky ReLU), parametric rectified linear unit (PReLU) and a new randomized leaky rectified linear units (RReLU). We evaluate these activation function on standard image classification task. Our experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results. Thus our findings are negative on the common belief that sparsity is the key of good performance in ReLU. Moreover, on small scale dataset, using deterministic negative slope or learning it are both prone to overfitting. They are not as effective as using their randomized counterpart. By using RReLU, we achieved 75.68% accuracy on CIFAR-100 test set without multiple test or ensemble.

Key facts. Cited by 519 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 May 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 1; 2021: 147; 2022: 103; 2023: 103; 2024: 81; 2025: 65; 2026: 17.

519 papers.

Sort:

By area:

  1. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  2. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 155
  3. 2021Hyperbolic Deep Neural Networks: A Surveycited by 139
  4. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
  5. 2021You Only Look One-level Featurecited by 116
  6. 2021Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learningcited by 106
  7. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  8. 2023Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Modelscited by 94
  9. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  10. 2021Sample Selection with Uncertainty of Losses for Learning with Noisy Labelscited by 79
  11. 2021On Transferability of Prompt Tuning for Natural Language Processingcited by 75
  12. 2023Multi-View Causal Representation Learning with Partial Observabilitycited by 74
38 more
  1. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70
  2. 2022MetaPortrait: Identity-Preserving Talking Head Generation with Fast Personalized Adaptationcited by 69
  3. 2023HeadGaS: Real-Time Animatable Head Avatars via 3D Gaussian Splattingcited by 60
  4. 2021PSA-GAN: Progressive Self Attention GANs for Synthetic Time Seriescited by 58
  5. 2021DPFM: Deep Partial Functional Mapscited by 58
  6. 2021Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifierscited by 57
  7. 20213D-to-2D Distillation for Indoor Scene Parsingcited by 48
  8. 2023Vision Transformer with Quadrangle Attentioncited by 46
  9. 2022VSA: Learning Varied-Size Window Attention in Vision Transformerscited by 43
  10. 2022Adversarial Robustness of Deep Neural Networks: A Survey from a Formal Verification Perspectivecited by 42
  11. 2021Shapley Explanation Networkscited by 41
  12. 2021Sparse Attention with Linear Unitscited by 41
  13. 2023FastPillars: A Deployment-friendly Pillar-based 3D Detectorcited by 39
  14. 2023Geometric Clifford Algebra Networkscited by 38
  15. 2021A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representationscited by 37
  16. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  17. 2021Single Image Depth Prediction with Wavelet Decompositioncited by 34
  18. 2021TabFairGAN: Fair Tabular Data Generation with Generative Adversarial Networkscited by 34
  19. 2023Picture that Sketch: Photorealistic Image Generation from Abstract Sketchescited by 34
  20. 2021Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvaturecited by 32
  21. 2021MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modelingcited by 32
  22. 2022Constrained Monotonic Neural Networkscited by 32
  23. 2021How to Design a Three-Stage Architecture for Audio-Visual Active Speaker Detection in the Wildcited by 31
  24. 2021Unsupervised Multi-Source Domain Adaptation for Person Re-Identificationcited by 30
  25. 2022NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilerscited by 30
  26. 2021Few-shot Open-set Recognition by Transformation Consistencycited by 29
  27. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  28. 2022Scalable Neural Video Representations with Learnable Positional Featurescited by 29
  29. 2023Understanding the Feature Norm for Out-of-Distribution Detectioncited by 27
  30. 2021Contextual Dropout: An Efficient Sample-Dependent Dropout Modulecited by 26
  31. 2021Dual Path Learning for Domain Adaptation of Semantic Segmentationcited by 26
  32. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  33. 2023Directions of Curvature as an Explanation for Loss of Plasticitycited by 26
  34. 2021Fusion-FlowNet: Energy-Efficient Optical Flow Estimation using Sensor Fusion and Deep Fused Spiking-Analog Network Architecturescited by 25
  35. 2021MS-MDA: Multisource Marginal Distribution Adaptation for Cross-subject and Cross-session EEG Emotion Recognitioncited by 25
  36. 2022HALSIE: Hybrid Approach to Learning Segmentation by Simultaneously Exploiting Image and Event Modalitiescited by 25
  37. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 25
  38. 2023Toward Understanding Generative Data Augmentationcited by 25

50 most cited of 519 All 519 (JSON)