mlsanityData to 7 Oct 2026

Large Scale Structure of Neural Network Loss Landscapes

arXiv 1906.04724 ↗First posted 11 Jun 2019cs.LG stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

There are many surprising and perhaps counter-intuitive properties of optimization of deep neural networks. We propose and experimentally verify a unified phenomenological model of the loss landscape that incorporates many of them. High dimensionality plays a key role in our model. Our core idea is to model the loss landscape as a set of high dimensional wedges that together form a large-scale, inter-connected structure and towards which optimization is drawn. We first show that hyperparameter choices such as learning rate, network width and L2 regularization, affect the path optimizer takes through the landscape in a similar ways, influencing the large scale curvature of the regions the optimizer explores. Finally, we predict and demonstrate new counter-intuitive properties of the loss-landscape. We show an existence of low loss subspaces connecting a set (not only a pair) of solutions, and verify it experimentally. Finally, we analyze recently popular ensembling techniques for deep networks in the light of our model.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 13; 2023: 6; 2024: 10; 2025: 9; 2026: 12.

68 papers.

Sort:

By area:

  1. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  2. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  3. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  4. 2023Deep Model Fusion: A Surveycited by 89
  5. 2021Learning Neural Network Subspacescited by 82
  6. 2022When Do Flat Minima Optimizers Work?cited by 78
  7. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  8. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  9. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  10. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  11. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  12. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
38 more
  1. 2021Towards Better Plasticity-Stability Trade-off in Incremental Learning: A Simple Linear Connectorcited by 34
  2. 2022A Closer Look at Learned Optimization: Stability, Robustness, and Inductive Biasescited by 30
  3. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  4. 2021Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapescited by 27
  5. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 27
  6. 2021Selfish Sparse RNN Trainingcited by 26
  7. 2024CreINNs: Credal-Set Interval Neural Networks for Uncertainty Estimation in Classification Taskscited by 26
  8. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  9. 2021Connecting Low-Loss Subspace for Personalized Federated Learningcited by 12
  10. 2023A Symmetry-Aware Exploration of Bayesian Neural Network Posteriorscited by 12
  11. 2021When Are Solutions Connected in Deep Networks?cited by 10
  12. 2026Low-Dimensional and Transversely Curved Optimization Dynamics in Grokkingcited by 10
  13. 2026The Geometry of Multi-Task Grokking: Transverse Instability, Superposition, and Weight Decay Phase Structurecited by 8
  14. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  15. 2022Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Trainingcited by 6
  16. 2024Dynamical stability and chaos in artificial neural network trajectories along trainingcited by 6
  17. 2025Paths and Ambient Spaces in Neural Loss Landscapescited by 6
  18. 2022What does a deep neural network confidently perceive? The effective dimension of high certainty class manifolds and their low confidence boundariescited by 5
  19. 2025Epistemic Wrapping for Uncertainty Quantificationcited by 5
  20. 2022Low-Loss Subspace Compression for Clean Gains against Multi-Agent Backdoor Attackscited by 4
  21. 2022Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysiscited by 4
  22. 2024Input Space Mode Connectivity in Deep Neural Networkscited by 4
  23. 2026Cut Less, Fold More: Model Compression through the Lens of Projection Geometrycited by 4
  24. 2022Interpolating Compressed Parameter Subspacescited by 3
  25. 2022A survey of deep learning optimizers -- first and second order methodscited by 3
  26. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  27. 2025Connecting Independently Trained Modes via Layer-Wise Connectivitycited by 3
  28. 2025Flat Channels to Infinity in Neural Loss Landscapescited by 3
  29. 2026Grokking as a Falsifiable Finite-Size Transitioncited by 3
  30. 2021Empirically explaining SGD from a line search perspectivecited by 2
  31. 2022Multiple Modes for Continual Learningcited by 2
  32. 2022Looking at the posterior: accuracy and uncertainty of neural-network predictionscited by 2
  33. 2023Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruningcited by 2
  34. 2023Consistent Explanations in the Face of Model Indeterminacy via Ensemblingcited by 2
  35. 2024Enhancing material property prediction with ensemble deep graph convolutional networkscited by 2
  36. 2025Analyzing the Role of Permutation Invariance in Linear Mode Connectivitycited by 2
  37. 2021Itsy Bitsy SpiderNet: Fully Connected Residual Network for Fraud Detectioncited by 1
  38. 2022Reinforcement Teachingcited by 1

50 most cited of 68 All 68 (JSON)