mlsanityData to 14 Oct 2026

Deep neural networks have an inbuilt Occam's razor

arXiv 2304.06670 ↗First posted 13 Apr 2023cs.LG cs.AI stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

The remarkable performance of overparameterized deep neural networks (DNNs) must arise from an interplay between network architecture, training algorithms, and structure in the data. To disentangle these three components, we apply a Bayesian picture, based on the functions expressed by a DNN, to supervised learning. The prior over functions is determined by the network, and is varied by exploiting a transition between ordered and chaotic regimes. For Boolean function classification, we approximate the likelihood using the error spectrum of functions on data. When combined with the prior, this accurately predicts the posterior, measured for DNNs trained with stochastic gradient descent. This analysis reveals that structured data, combined with an intrinsic Occam's razor-like inductive bias towards (Kolmogorov) simple functions that is strong enough to counteract the exponential growth of the number of functions with complexity, is a key to the success of DNNs.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 24 of its 59 references that are in the map. First posted on arXiv 13 Apr 2023.

Builds on

24 of 59 references.

References · 24

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  2. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  3. 2017Deep Neural Networks as Gaussian Processescited by 622
  4. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  5. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 351
  6. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  7. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  8. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 204
  9. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  10. 2016Deep Information Propagationcited by 195
  11. 2018To understand deep learning we need to understand kernel learningcited by 187
  12. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 183
12 more
  1. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  2. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  3. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
  4. 2018A high-bias, low-variance introduction to Machine Learning for physicistscited by 108
  5. 2016Why does deep and cheap learning work so well?cited by 100
  6. 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 96
  7. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 70
  8. 2019A Fine-Grained Spectral Perspective on Neural Networkscited by 68
  9. 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
  10. 2020Generalization bounds for deep learningcited by 38
  11. 2021Double-descent curves in neural networks: a new perspective using Gaussian processescited by 5
  12. 2022Investigating Generalization by Controlling Normalized Margincited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 5; 2025: 11; 2026: 10.

28 papers.

Sort:

By area:

  1. 2023In-Context Learning through the Bayesian Prismcited by 83
  2. 2024Learning Universal Predictorscited by 31
  3. 2025Block-Recurrent Dynamics in Vision Transformerscited by 20
  4. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 15
  5. 2025Parameter Symmetry Potentially Unifies Deep Learning Theorycited by 12
  6. 2025How Causal Abstraction Underpins Computational Explanationcited by 11
  7. 2024In-context learning and Occam's razorcited by 8
  8. 2025Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformerscited by 6
  9. 2024Do Quantum Neural Networks have Simplicity Bias?cited by 3
  10. 2024Exploiting the equivalence between quantum neural networks and perceptronscited by 3
  11. 2024The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labelscited by 3
  12. 2025Information-Theoretic Framework for Understanding Modern Machine-Learningcited by 3
16 more
  1. 2025A Minimum Description Length Approach to Regularization in Neural Networkscited by 2
  2. 2026Solomonoff inductioncited by 2
  3. 2025Characterising the Inductive Biases of Neural Networks on Boolean Datacited by 1
  4. 2026Does Order Matter : Connecting The Law of Robustness to Robust Generalizationcited by 1
  5. 2026Golden Handcuffs make safer AI agentscited by 1
  6. 2026Revisiting the Volume Hypothesiscited by 1
  7. 2023Simplicity bias, algorithmic probability, and the random logistic mapcited by 0
  8. 2025ResNets Are Deeper Than You Thinkcited by 0
  9. 2025Robust Deep Network Learning of Nonlinear Regression Tasks by Parametric Leaky Exponential Linear Units (LELUs) and a Diffusion Metriccited by 0
  10. 2025Polymorphic Combinatorial Frameworks (PCF): Guiding the Design of Mathematically-Grounded, Adaptive AI Agentscited by 0
  11. 2026BEACONS: Bounded-Error, Algebraically-Composable Neural Solvers for Partial Differential Equationscited by 0
  12. 2026Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Biascited by 0
  13. 2026Understanding Large Language Modelscited by 0
  14. 2026Benign interpolation and Occam's razorcited by 0
  15. 2026Dimensionless Controls of Plasticity Under Alternating Tasks: From Evolutionary Biology to Continual Learningcited by 0
  16. 2026Understanding Generalization Requires Universal Inductioncited by 0