# An empirical analysis of the optimization of deep network loss surfaces

arXiv: https://arxiv.org/abs/1612.04010 · HTML: https://mlsanity.com/paper/1612.04010/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2016.

## Built on by (most cited first, 21 of 21)

- [Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grok](https://mlsanity.com/paper/2306.13253/) (2023)
- [Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes](https://mlsanity.com/paper/2104.11044/) (2021)
- [Demystifying Batch Normalization in ReLU Networks: Equivalent Convex Optimization Models and Implicit Regularization](https://mlsanity.com/paper/2103.01499/) (2021)
- [Exploring the Geometry and Topology of Neural Network Loss Landscapes](https://mlsanity.com/paper/2102.00485/) (2021)
- [Data efficiency and extrapolation trends in neural network interatomic potentials](https://mlsanity.com/paper/2302.05823/) (2023)
- [Bidirectional Looking with A Novel Double Exponential Moving Average to Adaptive and Non-adaptive Momentum Optimizers](https://mlsanity.com/paper/2307.00631/) (2023)
- [Reevaluating Theoretical Analysis Methods for Optimization in Deep Learning](https://mlsanity.com/paper/2407.01825/) (2024)
- [Convex Dominance in Deep Learning I: A Scaling Law of Loss and Learning Rate](https://mlsanity.com/paper/2602.07145/) (2026)
- ["Reading Between the Heat": Co-Teaching Body Thermal Signatures for Non-intrusive Stress Detection](https://mlsanity.com/paper/2310.09932/) (2023)
- [Visualizing Loss Functions as Topological Landscape Profiles](https://mlsanity.com/paper/2411.12136/) (2024)
- [LossLens: Diagnostics for Machine Learning through Loss Landscape Visual Analytics](https://mlsanity.com/paper/2412.13321/) (2024)
- [Class Interference of Deep Neural Networks](https://mlsanity.com/paper/2211.01370/) (2022)
- [On-the-Fly Guidance Training for Medical Image Registration](https://mlsanity.com/paper/2308.15216/) (2023)
- [Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks](https://mlsanity.com/paper/2405.12489/) (2024)
- [Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks](https://mlsanity.com/paper/2405.12493/) (2024)
- [The effects of Hessian eigenvalue spectral density type on the applicability of Hessian analysis to generalization capability assessment of neural networks](https://mlsanity.com/paper/2504.17618/) (2025)
- [Visualizing the loss landscapes of physics-informed neural networks](https://mlsanity.com/paper/2602.05849/) (2026)
- [Landscaper: Understanding Loss Landscapes Through Multi-Dimensional Topological Analysis](https://mlsanity.com/paper/2602.07135/) (2026)
- [The layer-wise L1 Loss Landscape of Neural Nets is more complex around local minima](https://mlsanity.com/paper/2105.02831/) (2021)
- [On the Distributional Properties of Adaptive Gradients](https://mlsanity.com/paper/2105.07222/) (2021)
- [Visualization and Analysis of the Loss Landscape in Graph Neural Networks](https://mlsanity.com/paper/2509.11792/) (2025)

## Abstract (arXiv)

The success of deep neural networks hinges on our ability to accurately and efficiently optimize high-dimensional, non-convex functions. In this paper, we empirically investigate the loss functions of state-of-the-art networks, and how commonly-used stochastic gradient descent variants optimize these loss functions. To do this, we visualize the loss function by projecting them down to low-dimensional spaces chosen based on the convergence points of different optimization algorithms. Our observations suggest that optimization algorithms encounter and choose different descent directions at many saddle points to find different final weights. Based on consistency we observe across re-runs of the same stochastic optimization algorithm, we hypothesize that each optimization algorithm makes characteristic choices at these saddle points.