mlsanityData to 14 Oct 2026

Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks

arXiv 2201.07934 ↗First posted 3 Jan 2022cs.LG physics.bio-ph physics.data-ancited by 0

Not on the map. Why?

Abstract (arXiv)

An antithetical concept, adaptive symmetry, to conservative symmetry in physics is proposed to understand the deep neural networks (DNNs). It characterizes the invariance of variance, where a biotic system explores different pathways of evolution with equal probability in absence of feedback signals, and complex functional structure emerges from quantitative accumulation of adaptive-symmetries breaking in response to feedback signals. Theoretically and experimentally, we characterize the optimization process of a DNN system as an extended adaptive-symmetry-breaking process. One particular finding is that a hierarchically large DNN would have a large reservoir of adaptive symmetries, and when the information capacity of the reservoir exceeds the complexity of the dataset, the system could absorb all perturbations of the examples and self-organize into a functional structure of zero training errors measured by a certain surrogate risk. More specifically, this process is characterized by a statistical-mechanical model that could be appreciated as a generalization of statistics physics to the DNN organized complex system, and characterizes regularities in higher dimensionality. The model consists of three constitutes that could be appreciated as the counterparts of Boltzmann distribution, Ising model, and conservative symmetry, respectively: (1) a stochastic definition/interpretation of DNNs that is a multilayer probabilistic graphical model, (2) a formalism of circuits that perform biological computation, (3) a circuit symmetry from which self-similarity between the microscopic and the macroscopic adaptability manifests. The model is analyzed with a method referred as the statistical assembly method that analyzes the coarse-grained behaviors (over a symmetry group) of the heterogeneous hierarchical many-body interaction in DNNs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 132 of its 389 references that are in the map. First posted on arXiv 3 Jan 2022.

Builds on

132 of 389 references.

References · 132

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2013Intriguing properties of neural networkscited by 5,155
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  9. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  12. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
38 more
  1. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
  2. 2016Group Equivariant Convolutional Networkscited by 1,165
  3. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  4. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  6. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  7. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  8. 2016The Mythos of Model Interpretabilitycited by 721
  9. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  10. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  11. 2017Deep Neural Networks as Gaussian Processescited by 622
  12. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 588
  13. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  14. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  15. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  16. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  17. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 370
  18. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
  19. 2016Steerable CNNscited by 305
  20. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
  21. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  22. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  23. 2014The Loss Surfaces of Multilayer Networkscited by 280
  24. 2019Gauge Equivariant Convolutional Networks and the Icosahedral CNNcited by 267
  25. 2019Tackling Climate Change with Machine Learningcited by 265
  26. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  27. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  28. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  29. 2016Deep Learning without Poor Local Minimacited by 234
  30. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
  31. 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 187
  32. 2016Eigenvalues of the Hessian in Deep Learning: Singularity and Beyondcited by 159
  33. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  34. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  35. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  36. 2018Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batchescited by 133
  37. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  38. 2020Bayesian Neural Networks: An Introduction and Surveycited by 131

50 most cited of 132 All 132 (JSON)

Built on by

Not cited yet.