# Empirical Loss Landscape Analysis of Neural Network Activation Functions

arXiv: https://arxiv.org/abs/2306.16090 · HTML: https://mlsanity.com/paper/2306.16090/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 28 Jun 2023.

## Builds on (most cited first, 11 of 11)

- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Understanding Deep Neural Networks with Rectified Linear Units](https://mlsanity.com/paper/1611.01491/) (2016)
- [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017)
- [Understanding the Loss Surface of Neural Networks for Binary Classification](https://mlsanity.com/paper/1803.00909/) (2018)
- [Piecewise Strong Convexity of Neural Networks](https://mlsanity.com/paper/1810.12805/) (2018)
- [Visualising Basins of Attraction for the Cross-Entropy and the Squared Error Neural Network Loss Functions](https://mlsanity.com/paper/1901.02302/) (2019)
- [The Multilinear Structure of ReLU Networks](https://mlsanity.com/paper/1712.10132/) (2017)
- [Piecewise convexity of artificial neural networks](https://mlsanity.com/paper/1607.04917/) (2016)
- [Loss Surface Modality of Feed-Forward Neural Network Architectures](https://mlsanity.com/paper/1905.10268/) (2019)
- [The Loss Surface of XOR Artificial Neural Networks](https://mlsanity.com/paper/1804.02411/) (2018)

## Built on by (most cited first, 2 of 2)

- [Transformer-Based Approaches for Sensor-Based Human Activity Recognition: Opportunities and Challenges](https://mlsanity.com/paper/2410.13605/) (2024)
- [Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers](https://mlsanity.com/paper/2603.07122/) (2026)

## Abstract (arXiv)

Activation functions play a significant role in neural network design by enabling non-linearity. The choice of activation function was previously shown to influence the properties of the resulting loss landscape. Understanding the relationship between activation functions and loss landscape properties is important for neural architecture and training algorithm design. This study empirically investigates neural network loss landscapes associated with hyperbolic tangent, rectified linear unit, and exponential linear unit activation functions. Rectified linear unit is shown to yield the most convex loss landscape, and exponential linear unit is shown to yield the least flat loss landscape, and to exhibit superior generalisation performance. The presence of wide and narrow valleys in the loss landscape is established for all activation functions, and the narrow valleys are shown to correlate with saturated neurons and implicitly regularised network configurations.