mlsanityData to 14 Oct 2026

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

arXiv 1511.07289 ↗First posted 23 Nov 2015cs.LGcited by 1,070

Not on the map. Why?

Cited by 34 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce the "exponential linear unit" (ELU) which speeds up learning in deep neural networks and leads to higher classification accuracies. Like rectified linear units (ReLUs), leaky ReLUs (LReLUs) and parametrized ReLUs (PReLUs), ELUs alleviate the vanishing gradient problem via the identity for positive values. However, ELUs have improved learning characteristics compared to the units with other activation functions. In contrast to ReLUs, ELUs have negative values which allows them to push mean unit activations closer to zero like batch normalization but with lower computational complexity. Mean shifts toward zero speed up learning by bringing the normal gradient closer to the unit natural gradient because of a reduced bias shift effect. While LReLUs and PReLUs have negative values, too, they do not ensure a noise-robust deactivation state. ELUs saturate to a negative value with smaller inputs and thereby decrease the forward propagated variation and information. Therefore, ELUs code the degree of presence of particular phenomena in the input, while they do not quantitatively model the degree of their absence. In experiments, ELUs lead not only to faster learning, but also to significantly better generalization performance than ReLUs and LReLUs on networks with more than 5 layers. On CIFAR-100 ELUs networks significantly outperform ReLU networks with batch normalization while batch normalization does not improve ELU networks. ELU networks are among the top 10 reported CIFAR-10 results and yield the best published result on CIFAR-100, without resorting to multi-view evaluation or model averaging. On ImageNet, ELU networks considerably speed up learning compared to a ReLU network with the same architecture, obtaining less than 10% classification error for a single crop, single model network.

Key facts. Cited by 1,070 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 1; 2020: 2; 2021: 249; 2022: 192; 2023: 211; 2024: 190; 2025: 147; 2026: 77.

1,070 papers.

Sort:

By area:

  1. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,043
  2. 2022Building Normalizing Flows with Stochastic Interpolantscited by 995
  3. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  4. 2022High Fidelity Neural Audio Compressioncited by 687
  5. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  6. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  7. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  8. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  9. 2022Message Passing Neural PDE Solverscited by 386
  10. 2021Random Feature Attentioncited by 354
  11. 2021Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methodscited by 311
  12. 2022AudioGen: Textually Guided Audio Generationcited by 295
38 more
  1. 2021SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retrainingcited by 237
  2. 2021Weisfeiler and Lehman Go Topological: Message Passing Simplicial Networkscited by 221
  3. 2021Weisfeiler and Lehman Go Cellular: CW Networkscited by 218
  4. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  5. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 200
  6. 2021Large-Scale Representation Learning on Graphs via Bootstrappingcited by 186
  7. 2021Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networkscited by 176
  8. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  9. 2021A System for General In-Hand Object Re-Orientationcited by 165
  10. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 155
  11. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 149
  12. 2021MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesiscited by 133
  13. 2022Robust Collaborative 3D Object Detection in Presence of Pose Errorscited by 129
  14. 2024WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modelingcited by 129
  15. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 124
  16. 2020Primal Wasserstein Imitation Learningcited by 121
  17. 2022In-Hand Object Rotation via Rapid Motor Adaptationcited by 118
  18. 2021TransMVSNet: Global Context-aware Multi-view Stereo Network with Transformerscited by 117
  19. 2021Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimationcited by 116
  20. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 115
  21. 2021Global Self-Attention as a Replacement for Graph Convolutioncited by 114
  22. 2022Generative Flow Networks for Discrete Probabilistic Modelingcited by 114
  23. 2021APS: Active Pretraining with Successor Featurescited by 112
  24. 2021SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networkscited by 110
  25. 2022MonoDTR: Monocular 3D Object Detection with Depth-Aware Transformercited by 109
  26. 2022P3Depth: Monocular Depth Estimation with a Piecewise Planarity Priorcited by 107
  27. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 107
  28. 2022How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervisioncited by 102
  29. 2022Visual Dexterity: In-Hand Reorientation of Novel and Complex Object Shapescited by 101
  30. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  31. 2021Luna: Linear Unified Nested Attentioncited by 96
  32. 2023ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Modelscited by 94
  33. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  34. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  35. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 87
  36. 2022What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoderscited by 82
  37. 2021New Benchmarks for Learning on Non-Homophilous Graphscited by 81
  38. 2023Decoding Natural Images from EEG for Object Recognitioncited by 81

50 most cited of 1,070 All 1,070 (JSON)