mlsanityData to 7 Oct 2026

The semantic landscape paradigm for neural networks

arXiv 2307.09550 ↗First posted 18 Jul 2023cs.LG cond-mat.dis-nn cond-mat.stat-mechcited by 4

Not on the map. Why?

Abstract (arXiv)

Deep neural networks exhibit a fascinating spectrum of phenomena ranging from predictable scaling laws to the unpredictable emergence of new capabilities as a function of training time, dataset size and network size. Analysis of these phenomena has revealed the existence of concepts and algorithms encoded within the learned representations of these networks. While significant strides have been made in explaining observed phenomena separately, a unified framework for understanding, dissecting, and predicting the performance of neural networks is lacking. Here, we introduce the semantic landscape paradigm, a conceptual and mathematical framework that describes the training dynamics of neural networks as trajectories on a graph whose nodes correspond to emergent algorithms that are instrinsic to the learned representations of the networks. This abstraction enables us to describe a wide range of neural network phenomena in terms of well studied problems in statistical physics. Specifically, we show that grokking and emergence with scale are associated with percolation phenomena, and neural scaling laws are explainable in terms of the statistics of random walks on graphs. Finally, we discuss how the semantic landscape paradigm complements existing theoretical and practical approaches aimed at understanding and interpreting deep neural networks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 40 of its 59 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

40 of 59 references.

References · 40

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2021Scaling Vision Transformerscited by 993
  9. 2022In-context Learning and Induction Headscited by 930
  10. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  11. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
28 more
  1. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  2. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  3. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  4. 2021Explaining Neural Scaling Lawscited by 353
  5. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  6. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
  7. 2022Predictability and Surprise in Large Generative Modelscited by 209
  8. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 202
  9. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  10. 2021The Principles of Deep Learning Theorycited by 169
  11. 2023The Clock and the Pizza: Two Stories in Mechanistic Explanation of Neural Networkscited by 153
  12. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  13. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 142
  14. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 141
  15. 2023The Quantization Model of Neural Scalingcited by 137
  16. 2018A high-bias, low-variance introduction to Machine Learning for physicistscited by 108
  17. 2022A Solvable Model of Neural Scaling Lawscited by 101
  18. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  19. 2022The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenoncited by 90
  20. 2019Scaling description of generalization with number of parameters in deep learningcited by 83
  21. 2021Learning Curve Theorycited by 79
  22. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 78
  23. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  24. 2023Grokking of Hierarchical Structure in Vanilla Transformerscited by 53
  25. 2023Unifying Grokking and Double Descentcited by 49
  26. 2018Comparing Dynamics: Deep Neural Networks versus Glassy Systemscited by 42
  27. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38
  28. 2022Grokking phase transitions in learning local rules with gradient descentcited by 28

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  2. 2024A Resource Model For Neural Scaling Lawcited by 8
  3. 2025Information Gravity: A Field-Theoretic Model for Token Selection in Large Language Modelscited by 0
  4. 2025MonoCon: A general framework for learning ultra-compact high-fidelity representations using monotonicity constraintscited by 0