# Dataset-learning duality and emergent criticality

arXiv: https://arxiv.org/abs/2405.17391 · HTML: https://mlsanity.com/paper/2405.17391/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Mean Field View of the Landscape of Two-Layers Neural Networks](https://mlsanity.com/paper/1804.06561/) (2018)
- [Traditional and Heavy-Tailed Self Regularization in Neural Network Models](https://mlsanity.com/paper/1901.08276/) (2019)
- [Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks](https://mlsanity.com/paper/1901.08278/) (2019)
- [Eigenvalue distribution of nonlinear models of random matrices](https://mlsanity.com/paper/1904.03090/) (2019)
- [The world as a neural network](https://mlsanity.com/paper/2008.01540/) (2020)
- [Towards a Theory of Evolution as Multilevel Learning](https://mlsanity.com/paper/2110.14602/) (2021)
- [Towards a theory of machine learning](https://mlsanity.com/paper/2004.09280/) (2020)
- [Emergent Quantumness in Neural Networks](https://mlsanity.com/paper/2012.05082/) (2020)
- [Towards a theory of quantum gravity from neural networks](https://mlsanity.com/paper/2111.00903/) (2021)
- [Thermodynamics of Evolution and the Origin of Life](https://mlsanity.com/paper/2110.15066/) (2021)
- [Self-organized criticality in neural networks from activity-based rewiring](https://mlsanity.com/paper/2009.11781/) (2020)
- [Growing Critical: Self-Organized Criticality in a Developing Neural System](https://mlsanity.com/paper/1811.02861/) (2018)

## Built on by (most cited first, 5 of 5)

- [Emergent field theories from neural networks](https://mlsanity.com/paper/2411.08138/) (2024)
- [Covariant Gradient Descent](https://mlsanity.com/paper/2504.05279/) (2025)
- [Geometric Learning Dynamics](https://mlsanity.com/paper/2504.14728/) (2025)
- [Molecular Learning Dynamics](https://mlsanity.com/paper/2504.10560/) (2025)
- [Geometric framework for biological evolution](https://mlsanity.com/paper/2603.15198/) (2026)

## Abstract (arXiv)

In artificial neural networks, the activation dynamics of non-trainable variables is strongly coupled to the learning dynamics of trainable variables. During the activation pass, the boundary neurons (e.g., input neurons) are mapped to the bulk neurons (e.g., hidden neurons), and during the learning pass, both bulk and boundary neurons are mapped to changes in trainable variables (e.g., weights and biases). For example, in feed-forward neural networks, forward propagation is the activation pass and backward propagation is the learning pass. We show that a composition of the two maps establishes a duality map between a subspace of non-trainable boundary variables (e.g., dataset) and a tangent subspace of trainable variables (i.e., learning). In general, the dataset-learning duality is a complex non-linear map between high-dimensional spaces. We use duality to study the emergence of criticality, or the power-law distribution of fluctuations of the trainable variables, using a toy model at learning equilibrium. In particular, we show that criticality can emerge in the learning system even from the dataset in a non-critical state, and that the power-law distribution can be modified by changing either the activation function or the loss function.