# Towards a theory of machine learning

arXiv: https://arxiv.org/abs/2004.09280 · HTML: https://mlsanity.com/paper/2004.09280/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2020.

## Built on by (most cited first, 14 of 14)

- [Towards a Theory of Evolution as Multilevel Learning](https://mlsanity.com/paper/2110.14602/) (2021)
- [Towards a theory of quantum gravity from neural networks](https://mlsanity.com/paper/2111.00903/) (2021)
- [Thermodynamics of Evolution and the Origin of Life](https://mlsanity.com/paper/2110.15066/) (2021)
- [Quantum-Informed Machine Learning for Predicting Spatiotemporal Chaos with Practical Quantum Advantage](https://mlsanity.com/paper/2507.19861/) (2025)
- [Dataset-learning duality and emergent criticality](https://mlsanity.com/paper/2405.17391/) (2024)
- [Emergent field theories from neural networks](https://mlsanity.com/paper/2411.08138/) (2024)
- [Covariant Gradient Descent](https://mlsanity.com/paper/2504.05279/) (2025)
- [Geometric Learning Dynamics](https://mlsanity.com/paper/2504.14728/) (2025)
- [Self-organized criticality in neural networks](https://mlsanity.com/paper/2107.03402/) (2021)
- [Bio-inspired Machine Learning: programmed death and replication](https://mlsanity.com/paper/2207.04886/) (2022)
- [Molecular Learning Dynamics](https://mlsanity.com/paper/2504.10560/) (2025)
- [A method for quantifying the generalization capabilities of generative models for solving Ising models](https://mlsanity.com/paper/2405.03435/) (2024)
- [Verifying Good Regulator Conditions for Hypergraph Observers: Natural Gradient Learning from Causal Invariance via Established Theorems](https://mlsanity.com/paper/2603.09067/) (2026)
- [Geometric framework for biological evolution](https://mlsanity.com/paper/2603.15198/) (2026)

## Abstract (arXiv)

We define a neural network as a septuple consisting of (1) a state vector, (2) an input projection, (3) an output projection, (4) a weight matrix, (5) a bias vector, (6) an activation map and (7) a loss function. We argue that the loss function can be imposed either on the boundary (i.e. input and/or output neurons) or in the bulk (i.e. hidden neurons) for both supervised and unsupervised systems. We apply the principle of maximum entropy to derive a canonical ensemble of the state vectors subject to a constraint imposed on the bulk loss function by a Lagrange multiplier (or an inverse temperature parameter). We show that in an equilibrium the canonical partition function must be a product of two factors: a function of the temperature and a function of the bias vector and weight matrix. Consequently, the total Shannon entropy consists of two terms which represent respectively a thermodynamic entropy and a complexity of the neural network. We derive the first and second laws of learning: during learning the total entropy must decrease until the system reaches an equilibrium (i.e. the second law), and the increment in the loss function must be proportional to the increment in the thermodynamic entropy plus the increment in the complexity (i.e. the first law). We calculate the entropy destruction to show that the efficiency of learning is given by the Laplacian of the total free energy which is to be maximized in an optimal neural architecture, and explain why the optimization condition is better satisfied in a deep network with a large number of hidden layers. The key properties of the model are verified numerically by training a supervised feedforward neural network using the method of stochastic gradient descent. We also discuss a possibility that the entire universe on its most fundamental level is a neural network.