# Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective

arXiv: https://arxiv.org/abs/1906.05301 · HTML: https://mlsanity.com/paper/1906.05301/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 11 of 11)

- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functions](https://mlsanity.com/paper/2211.12316/) (2022)
- [Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNs](https://mlsanity.com/paper/2112.15383/) (2021)
- [A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs](https://mlsanity.com/paper/2106.04110/) (2021)
- [Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators](https://mlsanity.com/paper/2106.00694/) (2021)
- [A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs](https://mlsanity.com/paper/2105.01867/) (2021)
- [Mitigating the Curse of Detail: Scaling Arguments for Feature Learning and Sample Complexity](https://mlsanity.com/paper/2512.04165/) (2025)
- [Characterising the Inductive Biases of Neural Networks on Boolean Data](https://mlsanity.com/paper/2505.24060/) (2025)
- [Feature learning is decoupled from generalization in high capacity neural networks](https://mlsanity.com/paper/2507.19680/) (2025)
- [Complexity from Adaptive-Symmetries Breaking: Global Minima in the Statistical Mechanics of Deep Neural Networks](https://mlsanity.com/paper/2201.07934/) (2022)

## Abstract (arXiv)

In the past decade, deep neural networks (DNNs) came to the fore as the leading machine learning algorithms for a variety of tasks. Their raise was founded on market needs and engineering craftsmanship, the latter based more on trial and error than on theory. While still far behind the application forefront, the theoretical study of DNNs has recently made important advancements in analyzing the highly over-parameterized regime where some exact results have been obtained. Leveraging these ideas and adopting a more physics-like approach, here we construct a versatile field-theory formalism for supervised deep learning, involving renormalization group, Feynman diagrams and replicas. In particular we show that our approach leads to highly accurate predictions of learning curves of truly deep DNNs trained on polynomial regression tasks and that these predictions can be used for efficient hyper-parameter optimization. In addition, they explain how DNNs generalize well despite being highly over-parameterized, this due to an entropic bias to simple functions which, for the case of fully-connected DNNs with data sampled on the hypersphere, are low order polynomials in the input vector. Being a complex interacting system of artificial neurons, we believe that such tools and methodologies borrowed from condensed matter physics would prove essential for obtaining an accurate quantitative understanding of deep learning.