mlsanityData to 7 Oct 2026

Black holes and the loss landscape in machine learning

arXiv 2306.14817 ↗First posted 26 Jun 2023hep-th cs.LG stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Understanding the loss landscape is an important problem in machine learning. One key feature of the loss function, common to many neural network architectures, is the presence of exponentially many low lying local minima. Physical systems with similar energy landscapes may provide useful insights. In this work, we point out that black holes naturally give rise to such landscapes, owing to the existence of black hole entropy. For definiteness, we consider 1/8 BPS black holes in N = 8 string theory. These provide an infinite family of potential landscapes arising in the microscopic descriptions of corresponding black holes. The counting of minima amounts to black hole microstate counting. Moreover, the exact numbers of the minima for these landscapes are a priori known from dualities in string theory. Some of the minima are connected by paths of low loss values, resembling mode connectivity. We estimate the number of runs needed to find all the solutions. Initial explorations suggest that Stochastic Gradient Descent can find a significant fraction of the minima.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 47 of its 140 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

47 of 140 references.

References · 47

  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  3. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  4. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  5. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  6. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 286
  7. 2014The Loss Surfaces of Multilayer Networkscited by 278
  8. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  9. 2016Deep Learning without Poor Local Minimacited by 233
  10. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  11. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  12. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
35 more
  1. 2017Failures of Gradient-Based Deep Learningcited by 79
  2. 2017The loss surface of deep and wide neural networkscited by 74
  3. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  4. 2020Neural Networks and Quantum Field Theorycited by 58
  5. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
  6. 2020Learning to Unknotcited by 40
  7. 2020Neural Network Approximations for Calabi-Yau Metricscited by 32
  8. 2016No bad local minima: Data independent training error guarantees for multilayer neural networkscited by 29
  9. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  10. 2021Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlatorscited by 28
  11. 2021Building Quantum Field Theories Out of Neuronscited by 27
  12. 2021Nonperturbative renormalization for the neural network-QFT correspondencecited by 24
  13. 2021The edge of chaos: quantum field theory and deep neural networkscited by 23
  14. 2018Machine Learning CICY Threefoldscited by 22
  15. 2020Machine Learning for Condensed Matter Physicscited by 22
  16. 2017Deep-Learning the Landscapecited by 21
  17. 2018The loss surface of deep linear networks viewed through the algebraic geometry lenscited by 20
  18. 2014On the saddle point problem for non-convex optimizationcited by 19
  19. 2016Local minima in training of neural networkscited by 19
  20. 2020Quiver Mutations, Seiberg Duality and Machine Learningcited by 19
  21. 2020Machine learning for complete intersection Calabi-Yau manifolds: a methodological studycited by 17
  22. 2020Machine Learning Calabi-Yau Four-foldscited by 17
  23. 2021Machine Learning Calabi-Yau Hypersurfacescited by 15
  24. 2020Inception Neural Network for Complete Intersection Calabi-Yau 3-foldscited by 14
  25. 2022Machine Learned Calabi-Yau Metrics and Curvaturecited by 14
  26. 2020Machine Learning Lie Structures & Applications to Physicscited by 13
  27. 2023Structures of Neural Network Effective Theoriescited by 13
  28. 2020Disentangling a Deep Learned Volume Formulacited by 12
  29. 2021Learning knot invariants across dimensionscited by 12
  30. 2021Deep multi-task mining Calabi-Yau four-foldscited by 11
  31. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  32. 2022Renormalization in the neural network-quantum field theory correspondencecited by 7
  33. 2020Connecting Dualities and Machine Learningcited by 5
  34. 2020Sparsifying networks by traversing Geodesicscited by 1
  35. 2022Machine Learning Symmetrycited by 1

Built on by

Not cited yet.