mlsanityData to 14 Oct 2026

Continuously Differentiable Exponential Linear Units

arXiv 1704.07483 ↗First posted 24 Apr 2017cs.LGcited by 73

Not on the map. Why?

Abstract (arXiv)

Exponential Linear Units (ELUs) are a useful rectifier for constructing deep learning architectures, as they may speed up and otherwise improve learning by virtue of not have vanishing gradients and by having mean activations near zero. However, the ELU activation as parametrized in [1] is not continuously differentiable with respect to its input when the shape parameter alpha is not equal to 1. We present an alternative parametrization which is C1 continuous for all values of alpha, making the rectifier easier to reason about and making alpha easier to tune. This alternative parametrization has several other useful properties that the original parametrization of ELU does not: 1) its derivative with respect to x is bounded, 2) it contains both the linear transfer function and ReLU as special cases, and 3) it is scale-similar with respect to alpha.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 11; 2023: 15; 2024: 14; 2025: 17; 2026: 5.

73 papers.

Sort:

By area:

  1. 2021Activation Functions in Deep Learning: A Comprehensive Survey and Benchmarkcited by 155
  2. 2021Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmarkcited by 96
  3. 2023Smooth, exact rotational symmetrization for deep learning on point cloudscited by 39
  4. 2022Visual Attention Methods in Deep Learning: An In-Depth Surveycited by 37
  5. 2022On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Modelscited by 34
  6. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  7. 2022On amortizing convex conjugates for optimal transportcited by 31
  8. 2022How important are activation functions in regression and classification? A survey, performance comparison, and future directionscited by 29
  9. 2021Generative Video Transformer: Can Objects be the Words?cited by 24
  10. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
  11. 2021Machine learning nonequilibrium electron forces for adiabatic spin dynamicscited by 17
  12. 2023Self-Consistent Velocity Matching of Probability Flowscited by 17
38 more
  1. 2022Descriptors for Machine Learning Model of Generalized Force Field in Condensed Matter Systemscited by 16
  2. 2023A Framework for Demonstrating Practical Quantum Advantage: Racing Quantum against Classical Generative Modelscited by 16
  3. 2023Machine learning for phase ordering dynamics of charge density wavescited by 15
  4. 2025Learning Regularization Functionals for Inverse Problems: A Comparative Studycited by 15
  5. 2021AutoDO: Robust AutoAugment for Biased Data with Label Noise via Scalable Probabilistic Implicit Differentiationcited by 14
  6. 2021Towards Joint Intent Detection and Slot Filling via Higher-order Attentioncited by 11
  7. 2022Unsupervised Multi-object Segmentation Using Attention and Soft-argmaxcited by 11
  8. 2023Spatial Attention Kinetic Networks with E(n)-Equivariancecited by 11
  9. 2023Designing losses for data-free training of normalizing flows on Boltzmann distributionscited by 9
  10. 2024Kinetics of orbital ordering in cooperative Jahn-Teller models: Machine-learning enabled large-scale simulationscited by 9
  11. 2024Polynomial Composition Activations: Unleashing the Dynamics of Large Language Modelscited by 9
  12. 2025Learning of Population Dynamics: Inverse Optimization Meets JKO Schemecited by 9
  13. 2025Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activationscited by 8
  14. 2022Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networkscited by 6
  15. 2024Trainable Highly-expressive Activation Functionscited by 6
  16. 2025Activation Function Design Sustains Plasticity in Continual Learningcited by 6
  17. 2024Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimulicited by 5
  18. 2024A Method on Searching Better Activation Functionscited by 4
  19. 2025RBFleX-NAS: Training-Free Neural Architecture Search Using Radial Basis Function Kernel and Hyperparameter Detectioncited by 4
  20. 2022Synthetic data enable experiments in atomistic machine learningcited by 3
  21. 2023Bayesian Numerical Integration with Neural Networkscited by 3
  22. 2024Control of Medical Digital Twins with Artificial Neural Networkscited by 3
  23. 2021Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operatorscited by 2
  24. 2022Real World Large Scale Recommendation Systems Reproducibility and Smooth Activationscited by 2
  25. 2023Learn to Enhance the Negative Information in Convolutional Neural Networkcited by 2
  26. 2023Improving Resnet-9 Generalization Trained on Small Datasetscited by 2
  27. 2023Uncertainty Estimation in Contrast-Enhanced MR Image Translation with Multi-Axis Fusioncited by 2
  28. 2024Deep Continuous Networkscited by 2
  29. 2024Aligning Multiple Knowledge Graphs in a Single Passcited by 2
  30. 2025TakuNet: an Energy-Efficient CNN for Real-Time Inference on Embedded UAV systems in Emergency Response Scenarioscited by 2
  31. 2026Rational Neural Networks have Expressivity Advantagescited by 2
  32. 2022Beyond NaN: Resiliency of Optimization Layers in The Face of Infeasibilitycited by 1
  33. 2023Large-Batch, Iteration-Efficient Neural Bayesian Design Optimizationcited by 1
  34. 2023Machine learning force-field models for metallic spin glasscited by 1
  35. 2024Exploring the Relationship: Transformative Adaptive Activation Functions in Comparison to Other Activation Functionscited by 1
  36. 2025Training neural control variates using correlated configurationscited by 1
  37. 2021Computationally Efficient Multiscale Neural Networks Applied To Fluid Flow In Complex 3D Porous Mediacited by 0
  38. 2021HAN: Higher-order Attention Network for Spoken Language Understandingcited by 0

50 most cited of 73 All 73 (JSON)