# A continuum among logarithmic, linear, and exponential functions, and its potential to improve generalization in neural networks

arXiv: https://arxiv.org/abs/1602.01321 · HTML: https://mlsanity.com/paper/1602.01321/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Feb 2016.

## Built on by (most cited first, 3 of 3)

- [Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark](https://mlsanity.com/paper/2109.14545/) (2021)
- [Addressing Token Uniformity in Transformers via Singular Value Transformation](https://mlsanity.com/paper/2208.11790/) (2022)
- [MorphingNet: Symbolic Regression Where the Network Is the Formula](https://mlsanity.com/paper/2610.19942/) (2026)

## Abstract (arXiv)

We present the soft exponential activation function for artificial neural networks that continuously interpolates between logarithmic, linear, and exponential functions. This activation function is simple, differentiable, and parameterized so that it can be trained as the rest of the network is trained. We hypothesize that soft exponential has the potential to improve neural network learning, as it can exactly calculate many natural operations that typical neural networks can only approximate, including addition, multiplication, inner product, distance, polynomials, and sinusoids.