mlsanityData to 14 Oct 2026

Emergent weight morphologies in deep neural networks

arXiv 2501.05550 ↗First posted 9 Jan 2025cs.LG cond-mat.dis-nncited by 0

Not on the map. Why?

Abstract (arXiv)

Whether deep neural networks can exhibit emergent behaviour is not only relevant for understanding how deep learning works, it is also pivotal for estimating potential security risks of increasingly capable artificial intelligence systems. Here, we show that training deep neural networks gives rise to emergent weight morphologies independent of the training data. Specifically, in analogy to condensed matter physics, we derive a theory that predict that the homogeneous state of deep neural networks is unstable in a way that leads to the emergence of periodic channel structures. We verified these structures by performing numerical experiments on a variety of data sets. Our work demonstrates emergence in the training of deep neural networks, which impacts the achievable performance of deep neural networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

25 of 50 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2022Toy Models of Superpositioncited by 971
  8. 2015Deep Learning and the Information Bottleneck Principlecited by 802
  9. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  10. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  11. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 503
  12. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
13 more
  1. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  2. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 215
  3. 2022Predictability and Surprise in Large Generative Modelscited by 210
  4. 2023The Quantization Model of Neural Scalingcited by 138
  5. 2022Broken Neural Scaling Lawscited by 109
  6. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  7. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  8. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  9. 2015A Primer on Neural Network Models for Natural Language Processingcited by 76
  10. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 62
  11. 2020Neural Networks and Quantum Field Theorycited by 61
  12. 2014An exact mapping between the Variational Renormalization Group and Deep Learningcited by 56
  13. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42

Built on by

Not cited yet.