# The jamming transition as a paradigm to understand the loss landscape of deep neural networks

arXiv: https://arxiv.org/abs/1809.09349 · HTML: https://mlsanity.com/paper/1809.09349/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Sep 2018.

## Built on by (most cited first, 30 of 62)

- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [Scaling Laws and Interpretability of Learning from Repeated Data](https://mlsanity.com/paper/2205.10487/) (2022)
- [Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances](https://mlsanity.com/paper/2105.12221/) (2021)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Limitations of the NTK for Understanding Generalization in Deep Learning](https://mlsanity.com/paper/2206.10012/) (2022)
- [Taxonomizing local versus global structure in neural network loss landscapes](https://mlsanity.com/paper/2107.11228/) (2021)
- [Hessian Eigenspectra of More Realistic Nonlinear Models](https://mlsanity.com/paper/2103.01519/) (2021)
- [Multi-scale Feature Learning Dynamics: Insights for Double Descent](https://mlsanity.com/paper/2112.03215/) (2021)
- [Overparameterized Linear Regression under Adversarial Attacks](https://mlsanity.com/paper/2204.06274/) (2022)
- [Feature Learning and Signal Propagation in Deep Neural Networks](https://mlsanity.com/paper/2110.11749/) (2021)
- [Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning](https://mlsanity.com/paper/2101.08732/) (2021)
- [On double-descent in uncertainty quantification in overparametrized models](https://mlsanity.com/paper/2210.12760/) (2022)
- [On the interplay between data structure and loss function in classification problems](https://mlsanity.com/paper/2103.05524/) (2021)
- [Synthetic Face Datasets Generation via Latent Space Exploration from Brownian Identity Diffusion](https://mlsanity.com/paper/2405.00228/) (2024)
- [On the Lipschitz Constant of Deep Networks and Double Descent](https://mlsanity.com/paper/2301.12309/) (2023)
- [Edge of chaos as a guiding principle for modern neural network training](https://mlsanity.com/paper/2107.09437/) (2021)
- [Mitigating multiple descents: A model-agnostic framework for risk monotonization](https://mlsanity.com/paper/2205.12937/) (2022)
- [Models of Heavy-Tailed Mechanistic Universality](https://mlsanity.com/paper/2506.03470/) (2025)
- [Effects of noise on the overparametrization of quantum neural networks](https://mlsanity.com/paper/2302.05059/) (2023)
- [Generalization for multiclass classification with overparameterized linear models](https://mlsanity.com/paper/2206.01399/) (2022)
- [Deep Double Descent via Smooth Interpolation](https://mlsanity.com/paper/2209.10080/) (2022)
- [No Free Prune: Information-Theoretic Barriers to Pruning at Initialization](https://mlsanity.com/paper/2402.01089/) (2024)
- [Tilting the playing field: Dynamical loss functions for machine learning](https://mlsanity.com/paper/2102.03793/) (2021)
- [Distilling Double Descent](https://mlsanity.com/paper/2102.06849/) (2021)
- [How DNNs break the Curse of Dimensionality: Compositionality and Symmetry Learning](https://mlsanity.com/paper/2407.05664/) (2024)
- [Recurrences reveal shared causal drivers of complex time series](https://mlsanity.com/paper/2301.13516/) (2023)
- [Mitigating deep double descent by concatenating inputs](https://mlsanity.com/paper/2107.00797/) (2021)
- [Solution space and storage capacity of fully connected two-layer neural networks with generic activation functions](https://mlsanity.com/paper/2404.13404/) (2024)
- [Connecting Independently Trained Modes via Layer-Wise Connectivity](https://mlsanity.com/paper/2505.02604/) (2025)

## Abstract (arXiv)

Deep learning has been immensely successful at a variety of tasks, ranging from classification to AI. Learning corresponds to fitting training data, which is implemented by descending a very high-dimensional loss function. Understanding under which conditions neural networks do not get stuck in poor minima of the loss, and how the landscape of that loss evolves as depth is increased remains a challenge. Here we predict, and test empirically, an analogy between this landscape and the energy landscape of repulsive ellipses. We argue that in FC networks a phase transition delimits the over- and under-parametrized regimes where fitting can or cannot be achieved. In the vicinity of this transition, properties of the curvature of the minima of the loss are critical. This transition shares direct similarities with the jamming transition by which particles form a disordered solid as the density is increased, which also occurs in certain classes of computational optimization and learning problems such as the perceptron. Our analysis gives a simple explanation as to why poor minima of the loss cannot be encountered in the overparametrized regime, and puts forward the surprising result that the ability of fully connected networks to fit random data is independent of their depth. Our observations suggests that this independence also holds for real data. We also study a quantity Delta which characterizes how well (Delta<0) or badly (Delta>0) a datum is learned. At the critical point it is power-law distributed, P_+(Delta)~Deltaθ for Delta>0 and P_-(Delta)~(-Delta)-γ for Delta<0, with θ≈0.3 and γ≈0.2. This observation suggests that near the transition the loss landscape has a hierarchical structure and that the learning dynamics is prone to avalanche-like dynamics, with abrupt changes in the set of patterns that are learned.