# Loss Patterns of Neural Networks

arXiv: https://arxiv.org/abs/1910.03867 · HTML: https://mlsanity.com/paper/1910.03867/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2019.

## Built on by (most cited first, 10 of 10)

- [Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling](https://mlsanity.com/paper/2102.13042/) (2021)
- [Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes](https://mlsanity.com/paper/2204.11326/) (2022)
- [Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis](https://mlsanity.com/paper/2105.01650/) (2021)
- [Embedding Principle of Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2105.14573/) (2021)
- [Embedding Principle: a hierarchical structure of loss landscape of deep neural networks](https://mlsanity.com/paper/2111.15527/) (2021)
- [Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks](https://mlsanity.com/paper/2205.13283/) (2022)
- [LossPlot: A Better Way to Visualize Loss Landscapes](https://mlsanity.com/paper/2111.15133/) (2021)
- [Multi-attacks: Many images + the same adversarial attack → many target labels](https://mlsanity.com/paper/2308.03792/) (2023)
- [Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks](https://mlsanity.com/paper/2405.17501/) (2024)
- [Context-Adaptive Deep Neural Networks via Bridge-Mode Connectivity](https://mlsanity.com/paper/2211.15436/) (2022)

## Abstract (arXiv)

We present multi-point optimization: an optimization technique that allows to train several models simultaneously without the need to keep the parameters of each one individually. The proposed method is used for a thorough empirical analysis of the loss landscape of neural networks. By extensive experiments on FashionMNIST and CIFAR10 datasets we demonstrate two things: 1) loss surface is surprisingly diverse and intricate in terms of landscape patterns it contains, and 2) adding batch normalization makes it more smooth. Source code to reproduce all the reported results is available on GitHub: https://github.com/universome/loss-patterns.