Loss Patterns of Neural Networks
Not on the map. Why?
Abstract (arXiv)
We present multi-point optimization: an optimization technique that allows to train several models simultaneously without the need to keep the parameters of each one individually. The proposed method is used for a thorough empirical analysis of the loss landscape of neural networks. By extensive experiments on FashionMNIST and CIFAR10 datasets we demonstrate two things: 1) loss surface is surprisingly diverse and intricate in terms of landscape patterns it contains, and 2) adding batch normalization makes it more smooth. Source code to reproduce all the reported results is available on GitHub: https://github.com/universome/loss-patterns.
Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Oct 2019.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 5; 2022: 3; 2023: 1; 2024: 1.
10 papers.
- 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
- 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
- 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysiscited by 39
- 2021Embedding Principle of Loss Landscape of Deep Neural Networkscited by 38
- 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networkscited by 22
- 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networkscited by 7
- 2021LossPlot: A Better Way to Visualize Loss Landscapescited by 5
- 2023Multi-attacks: Many images + the same adversarial attack → many target labelscited by 3
- 2024Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networkscited by 2
- 2022Context-Adaptive Deep Neural Networks via Bridge-Mode Connectivitycited by 0
- 2024Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks26 May 2024
- 2023Multi-attacks: Many images + the same adversarial attack → many target labels4 Aug 2023
- 2022Context-Adaptive Deep Neural Networks via Bridge-Mode Connectivity28 Nov 2022
- 2022Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks26 May 2022
- 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapes24 Apr 2022
- 2021Embedding Principle: a hierarchical structure of loss landscape of deep neural networks30 Nov 2021
- 2021LossPlot: A Better Way to Visualize Loss Landscapes30 Nov 2021
- 2021Embedding Principle of Loss Landscape of Deep Neural Networks30 May 2021
- 2021Stochastic gradient descent with noise of machine learning type. Part I: Discrete time analysis4 May 2021
- 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling25 Feb 2021