mlsanityData to 14 Oct 2026

Comparing Dynamics: Deep Neural Networks versus Glassy Systems

arXiv 1803.06969 ↗First posted 19 Mar 2018stat.ML cond-mat.dis-nn cs.LGcited by 42

Not on the map. Why?

Abstract (arXiv)

We analyze numerically the training dynamics of deep neural networks (DNN) by using methods developed in statistical physics of glassy systems. The two main issues we address are (1) the complexity of the loss landscape and of the dynamics within it, and (2) to what extent DNNs share similarities with glassy systems. Our findings, obtained for different architectures and datasets, suggest that during the training process the dynamics slows down because of an increasingly large number of flat directions. At large times, when the loss is approaching zero, the system diffuses at the bottom of the landscape. Despite some similarities with the dynamics of mean-field glassy systems, in particular, the absence of barrier crossing, we find distinctive dynamical behaviors in the two cases, showing that the statistical properties of the corresponding loss and energy landscapes are different. In contrast, when the network is under-parametrized we observe a typical glassy behavior, thus suggesting the existence of different phases depending on whether the network is under-parametrized or over-parametrized.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 9; 2022: 4; 2023: 10; 2024: 5; 2025: 7; 2026: 7.

42 papers.

Sort:

By area:

  1. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
  2. 2021Exponentially Many Local Minima in Quantum Neural Networkscited by 22
  3. 2021Learning through atypical "phase transitions" in overparameterized neural networkscited by 21
  4. 2023On the Impact of Overparameterization on the Training of a Shallow Neural Network in High Dimensionscited by 21
  5. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 20
  6. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
  7. 2024Disentangling and Mitigating the Impact of Task Similarity for Continual Learningcited by 19
  8. 2023A Three-regime Model of Network Pruningcited by 15
  9. 2023Typical and atypical solutions in non-convex neural networks with discrete and continuous weightscited by 13
  10. 2021Edge of chaos as a guiding principle for modern neural network trainingcited by 10
  11. 2021Ensemble Inference Methods for Models With Noisy and Expensive Likelihoodscited by 8
  12. 2021Characterization of Generalizability of Spike Timing Dependent Plasticity trained Spiking Neural Networkscited by 8
30 more
  1. 2024The Role of the Time-Dependent Hessian in High-Dimensional Optimizationcited by 7
  2. 2022Quantifying Relevance in Learning and Inferencecited by 5
  3. 2021A spin-glass model for the loss surfaces of generative adversarial networkscited by 4
  4. 2022Universal characteristics of deep neural network loss surfaces from random matrix theorycited by 4
  5. 2023The semantic landscape paradigm for neural networkscited by 4
  6. 2023High-dimensional manifold of solutions in neural networks: insights from statistical physicscited by 4
  7. 2024Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noisecited by 4
  8. 2022Understanding out-of-distribution accuracies through quantifying difficulty of test samplescited by 3
  9. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  10. 2025The Effect of Label Noise on the Information Content of Neural Representationscited by 2
  11. 2025Overparametrization bends the landscape: BBP transitions at initialization in simple Neural Networkscited by 2
  12. 2026Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networkscited by 2
  13. 2021Mode connectivity in the loss landscape of parameterized quantum circuitscited by 1
  14. 2023Tradeoff of generalization error in unsupervised learningcited by 1
  15. 2023Evolutionary Algorithms in the Light of SGD: Limit Equivalence, Minima Flatness, and Transfer Learningcited by 1
  16. 2025Rethinking Over-Smoothing in Graph Neural Networks: A Perspective from Anderson Localizationcited by 1
  17. 2025Entropic Confinement and Mode Connectivity in Overparameterized Neural Networkscited by 1
  18. 2026Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrievalcited by 1
  19. 2026Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networkscited by 1
  20. 2026Unveiling Multi-regime Patterns in SciML: Distinct Failure Modes and Regime-specific Optimizationcited by 1
  21. 2026Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localizationcited by 1
  22. 2022Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networkscited by 0
  23. 2023Black holes and the loss landscape in machine learningcited by 0
  24. 2023Emergent learning in physical systems as feedback-based aging in a glassy landscapecited by 0
  25. 2024The twin peaks of learning neural networkscited by 0
  26. 2024Learnability of high-dimensional targets by two-parameter models and gradient flowcited by 0
  27. 2025Emergent weight morphologies in deep neural networkscited by 0
  28. 2025Categorical Invariants of Learning Dynamicscited by 0
  29. 2026High-Dimensional Latents Should Be Diagnosed Through Phase Structurecited by 0
  30. 2026Percolation Dynamics in Optimization: Variance Cascades and Nested Symmetrycited by 0