Statistical Properties of Training & Generalization
Not on the map. Why?
Abstract (arXiv)
Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 117 of its 160 references that are in the map. First posted on arXiv 18 Jun 2026.
Builds on
117 of 160 references.
References · 117
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
- 2017Deep Setscited by 1,421
38 more
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,167
- 2016Group Equivariant Convolutional Networkscited by 1,165
- 2017Mixed Precision Trainingcited by 1,156
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2018On Lazy Training in Differentiable Programmingcited by 675
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
- 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
- 2017Deep Neural Networks as Gaussian Processescited by 622
- 2014Convex Optimization: Algorithms and Complexitycited by 575
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
- 2019Benign Overfitting in Linear Regressioncited by 549
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
- 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 426
- 2021Explaining Neural Scaling Lawscited by 363
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 330
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
- 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
- 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
- 2017Stochastic Gradient Descent as Approximate Bayesian Inferencecited by 300
- 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
- 2014The Loss Surfaces of Multilayer Networkscited by 280
- 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
- 2017High-dimensional dynamics of generalization error in neural networkscited by 241
- 2016Deep Learning without Poor Local Minimacited by 234
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
- 2020The large learning rate phase of deep learning: the catapult mechanismcited by 216
- 2018To understand deep learning we need to understand kernel learningcited by 187
- 2021The Principles of Deep Learning Theorycited by 173
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
- 2020Benign overfitting in ridge regressioncited by 153
50 most cited of 117 All 117 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.