# Black holes and the loss landscape in machine learning

arXiv: https://arxiv.org/abs/2306.14817 · HTML: https://mlsanity.com/paper/2306.14817/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 47 of its 140 references that are in the map.
- First posted on arXiv 26 Jun 2023.

## Builds on (most cited first, 30 of 47)

- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs](https://mlsanity.com/paper/1802.10026/) (2018)
- [Entropy-SGD: Biasing Gradient Descent Into Wide Valleys](https://mlsanity.com/paper/1611.01838/) (2016)
- [Identifying and attacking the saddle point problem in high-dimensional non-convex optimization](https://mlsanity.com/paper/1406.2572/) (2014)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Train longer, generalize better: closing the generalization gap in large batch training of neural networks](https://mlsanity.com/paper/1705.08741/) (2017)
- [Deep Learning without Poor Local Minima](https://mlsanity.com/paper/1605.07110/) (2016)
- [Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks](https://mlsanity.com/paper/1811.08888/) (2018)
- [Spurious Local Minima are Common in Two-Layer ReLU Neural Networks](https://mlsanity.com/paper/1712.08968/) (2017)
- [Topology and Geometry of Half-Rectified Network Optimization](https://mlsanity.com/paper/1611.01540/) (2016)
- [Failures of Gradient-Based Deep Learning](https://mlsanity.com/paper/1703.07950/) (2017)
- [The loss surface of deep and wide neural networks](https://mlsanity.com/paper/1704.08045/) (2017)
- [On the Computational Efficiency of Training Neural Networks](https://mlsanity.com/paper/1410.1141/) (2014)
- [Neural Networks and Quantum Field Theory](https://mlsanity.com/paper/2008.08601/) (2020)
- [Comparing Dynamics: Deep Neural Networks versus Glassy Systems](https://mlsanity.com/paper/1803.06969/) (2018)
- [Learning to Unknot](https://mlsanity.com/paper/2010.16263/) (2020)
- [Neural Network Approximations for Calabi-Yau Metrics](https://mlsanity.com/paper/2012.15821/) (2020)
- [No bad local minima: Data independent training error guarantees for multilayer neural networks](https://mlsanity.com/paper/1605.08361/) (2016)
- [Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators](https://mlsanity.com/paper/2106.00694/) (2021)
- [Small nonlinearities in activation functions create bad local minima in neural networks](https://mlsanity.com/paper/1802.03487/) (2018)
- [Building Quantum Field Theories Out of Neurons](https://mlsanity.com/paper/2112.04527/) (2021)
- [Nonperturbative renormalization for the neural network-QFT correspondence](https://mlsanity.com/paper/2108.01403/) (2021)
- [The edge of chaos: quantum field theory and deep neural networks](https://mlsanity.com/paper/2109.13247/) (2021)
- [Machine Learning CICY Threefolds](https://mlsanity.com/paper/1806.03121/) (2018)
- [Machine Learning for Condensed Matter Physics](https://mlsanity.com/paper/2005.14228/) (2020)
- [Deep-Learning the Landscape](https://mlsanity.com/paper/1706.02714/) (2017)
- [On the saddle point problem for non-convex optimization](https://mlsanity.com/paper/1405.4604/) (2014)
- [The loss surface of deep linear networks viewed through the algebraic geometry lens](https://mlsanity.com/paper/1810.07716/) (2018)

## Abstract (arXiv)

Understanding the loss landscape is an important problem in machine learning. One key feature of the loss function, common to many neural network architectures, is the presence of exponentially many low lying local minima. Physical systems with similar energy landscapes may provide useful insights. In this work, we point out that black holes naturally give rise to such landscapes, owing to the existence of black hole entropy. For definiteness, we consider 1/8 BPS black holes in N = 8 string theory. These provide an infinite family of potential landscapes arising in the microscopic descriptions of corresponding black holes. The counting of minima amounts to black hole microstate counting. Moreover, the exact numbers of the minima for these landscapes are a priori known from dualities in string theory. Some of the minima are connected by paths of low loss values, resembling mode connectivity. We estimate the number of runs needed to find all the solutions. Initial explorations suggest that Stochastic Gradient Descent can find a significant fraction of the minima.