# Exploring the Geometry and Topology of Neural Network Loss Landscapes

arXiv: https://arxiv.org/abs/2102.00485 · HTML: https://mlsanity.com/paper/2102.00485/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 31 Jan 2021.

## Builds on (most cited first, 11 of 11)

- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Qualitatively characterizing neural network optimization problems](https://mlsanity.com/paper/1412.6544/) (2014)
- [Deep Learning with Topological Signatures](https://mlsanity.com/paper/1707.04041/) (2017)
- [Graph Filtration Learning](https://mlsanity.com/paper/1905.10996/) (2019)
- [Learning metrics for persistence-based summaries and applications for graph classification](https://mlsanity.com/paper/1904.12189/) (2019)
- [Uncovering the Topology of Time-Varying fMRI Data using Cubical Persistence](https://mlsanity.com/paper/2006.07882/) (2020)
- [An empirical analysis of the optimization of deep network loss surfaces](https://mlsanity.com/paper/1612.04010/) (2016)

## Built on by (most cited first, 14 of 14)

- [Position: Topological Deep Learning is the New Frontier for Relational Learning](https://mlsanity.com/paper/2402.08871/) (2024)
- [Visualizing high-dimensional loss landscapes with Hessian directions](https://mlsanity.com/paper/2208.13219/) (2022)
- [Near-optimal control of dynamical systems with neural ordinary differential equations](https://mlsanity.com/paper/2206.11120/) (2022)
- [Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis](https://mlsanity.com/paper/2407.05385/) (2024)
- [Diffusion Curvature for Estimating Local Curvature in High Dimensional Data](https://mlsanity.com/paper/2206.03977/) (2022)
- [Topological obstruction to the training of shallow ReLU neural networks](https://mlsanity.com/paper/2410.14837/) (2024)
- [Mapping the Multiverse of Latent Representations](https://mlsanity.com/paper/2402.01514/) (2024)
- [Bringing the Discussion of Minima Sharpness to the Audio Domain: a Filter-Normalised Evaluation for Acoustic Scene Classification](https://mlsanity.com/paper/2309.16369/) (2023)
- [Addressing caveats of neural persistence with deep graph persistence](https://mlsanity.com/paper/2307.10865/) (2023)
- [Visualizing the loss landscapes of physics-informed neural networks](https://mlsanity.com/paper/2602.05849/) (2026)
- [Neuro-Visualizer: An Auto-encoder-based Loss Landscape Visualization Method](https://mlsanity.com/paper/2309.14601/) (2023)
- [Statistical Mechanics and Artificial Neural Networks: Principles, Models, and Applications](https://mlsanity.com/paper/2405.10957/) (2024)
- [Non-Uniform Parameter-Wise Model Merging](https://mlsanity.com/paper/2412.15467/) (2024)
- [SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation](https://mlsanity.com/paper/2607.23346/) (2026)

## Abstract (arXiv)

Recent work has established clear links between the generalization performance of trained neural networks and the geometry of their loss landscape near the local minima to which they converge. This suggests that qualitative and quantitative examination of the loss landscape geometry could yield insights about neural network generalization performance during training. To this end, researchers have proposed visualizing the loss landscape through the use of simple dimensionality reduction techniques. However, such visualization methods have been limited by their linear nature and only capture features in one or two dimensions, thus restricting sampling of the loss landscape to lines or planes. Here, we expand and improve upon these in three ways. First, we present a novel "jump and retrain" procedure for sampling relevant portions of the loss landscape. We show that the resulting sampled data holds more meaningful information about the network's ability to generalize. Next, we show that non-linear dimensionality reduction of the jump and retrain trajectories via PHATE, a trajectory and manifold-preserving method, allows us to visualize differences between networks that are generalizing well vs poorly. Finally, we combine PHATE trajectories with a computational homology characterization to quantify trajectory differences.