# Continuously Differentiable Exponential Linear Units

arXiv: https://arxiv.org/abs/1704.07483 · HTML: https://mlsanity.com/paper/1704.07483/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Apr 2017.

## Built on by (most cited first, 30 of 73)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Do Neural Optimal Transport Solvers Work? A Continuous Wasserstein-2 Benchmark](https://mlsanity.com/paper/2106.01954/) (2021)
- [Smooth, exact rotational symmetrization for deep learning on point clouds](https://mlsanity.com/paper/2305.19302/) (2023)
- [Visual Attention Methods in Deep Learning: An In-Depth Survey](https://mlsanity.com/paper/2204.07756/) (2022)
- [On the Factory Floor: ML Engineering for Industrial-Scale Ads Recommendation Models](https://mlsanity.com/paper/2209.05310/) (2022)
- [Deep Network Approximation: Beyond ReLU to Diverse Activation Functions](https://mlsanity.com/paper/2307.06555/) (2023)
- [On amortizing convex conjugates for optimal transport](https://mlsanity.com/paper/2210.12153/) (2022)
- [How important are activation functions in regression and classification? A survey, performance comparison, and future directions](https://mlsanity.com/paper/2209.02681/) (2022)
- [Generative Video Transformer: Can Objects be the Words?](https://mlsanity.com/paper/2107.09240/) (2021)
- [HSR-Enhanced Sparse Attention Acceleration](https://mlsanity.com/paper/2410.10165/) (2024)
- [Machine learning nonequilibrium electron forces for adiabatic spin dynamics](https://mlsanity.com/paper/2112.12124/) (2021)
- [Self-Consistent Velocity Matching of Probability Flows](https://mlsanity.com/paper/2301.13737/) (2023)
- [Descriptors for Machine Learning Model of Generalized Force Field in Condensed Matter Systems](https://mlsanity.com/paper/2201.00798/) (2022)
- [A Framework for Demonstrating Practical Quantum Advantage: Racing Quantum against Classical Generative Models](https://mlsanity.com/paper/2303.15626/) (2023)
- [Machine learning for phase ordering dynamics of charge density waves](https://mlsanity.com/paper/2303.03493/) (2023)
- [Learning Regularization Functionals for Inverse Problems: A Comparative Study](https://mlsanity.com/paper/2510.01755/) (2025)
- [AutoDO: Robust AutoAugment for Biased Data with Label Noise via Scalable Probabilistic Implicit Differentiation](https://mlsanity.com/paper/2103.05863/) (2021)
- [Towards Joint Intent Detection and Slot Filling via Higher-order Attention](https://mlsanity.com/paper/2109.08890/) (2021)
- [Unsupervised Multi-object Segmentation Using Attention and Soft-argmax](https://mlsanity.com/paper/2205.13271/) (2022)
- [Spatial Attention Kinetic Networks with E(n)-Equivariance](https://mlsanity.com/paper/2301.08893/) (2023)
- [Designing losses for data-free training of normalizing flows on Boltzmann distributions](https://mlsanity.com/paper/2301.05475/) (2023)
- [Kinetics of orbital ordering in cooperative Jahn-Teller models: Machine-learning enabled large-scale simulations](https://mlsanity.com/paper/2405.14776/) (2024)
- [Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models](https://mlsanity.com/paper/2411.03884/) (2024)
- [Learning of Population Dynamics: Inverse Optimization Meets JKO Scheme](https://mlsanity.com/paper/2506.01502/) (2025)
- [Advancing Constrained Monotonic Neural Networks: Achieving Universal Approximation Beyond Bounded Activations](https://mlsanity.com/paper/2505.02537/) (2025)
- [Control of Dual-Sourcing Inventory Systems using Recurrent Neural Networks](https://mlsanity.com/paper/2201.06126/) (2022)
- [Trainable Highly-expressive Activation Functions](https://mlsanity.com/paper/2407.07564/) (2024)
- [Activation Function Design Sustains Plasticity in Continual Learning](https://mlsanity.com/paper/2509.22562/) (2025)
- [Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli](https://mlsanity.com/paper/2411.01505/) (2024)
- [A Method on Searching Better Activation Functions](https://mlsanity.com/paper/2405.12954/) (2024)

## Abstract (arXiv)

Exponential Linear Units (ELUs) are a useful rectifier for constructing deep learning architectures, as they may speed up and otherwise improve learning by virtue of not have vanishing gradients and by having mean activations near zero. However, the ELU activation as parametrized in [1] is not continuously differentiable with respect to its input when the shape parameter alpha is not equal to 1. We present an alternative parametrization which is C1 continuous for all values of alpha, making the rectifier easier to reason about and making alpha easier to tune. This alternative parametrization has several other useful properties that the original parametrization of ELU does not: 1) its derivative with respect to x is bounded, 2) it contains both the linear transfer function and ReLU as special cases, and 3) it is scale-similar with respect to alpha.