# The layer-wise L1 Loss Landscape of Neural Nets is more complex around local minima

arXiv: https://arxiv.org/abs/2105.02831 · HTML: https://mlsanity.com/paper/2105.02831/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 5 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 4 of 4)

- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [The loss surface of deep and wide neural networks](https://mlsanity.com/paper/1704.08045/) (2017)
- [An empirical analysis of the optimization of deep network loss surfaces](https://mlsanity.com/paper/1612.04010/) (2016)
- [Deep ReLU Programming](https://mlsanity.com/paper/2011.14895/) (2020)

## Abstract (arXiv)

For fixed training data and network parameters in the other layers the L1 loss of a ReLU neural network as a function of the first layer's parameters is a piece-wise affine function. We use the Deep ReLU Simplex algorithm to iteratively minimize the loss monotonically on adjacent vertices and analyze the trajectory of these vertex positions. We empirically observe that in a neighbourhood around a local minimum, the iterations behave differently such that conclusions on loss level and proximity of the local minimum can be made before it has been found: Firstly the loss seems to decay exponentially slow at iterated adjacent vertices such that the loss level at the local minimum can be estimated from the loss levels of subsequently iterated vertices, and secondly we observe a strong increase of the vertex density around local minima. This could have far-reaching consequences for the design of new gradient-descent algorithms that might improve convergence rate by exploiting these facts.