# A random energy approach to deep learning

arXiv: https://arxiv.org/abs/2112.09420 · HTML: https://mlsanity.com/paper/2112.09420/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 34 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 9 of 9)

- [Intrinsic dimension of data representations in deep neural networks](https://mlsanity.com/paper/1905.12784/) (2019)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Statistical physics of inference: Thresholds and algorithms](https://mlsanity.com/paper/1511.02476/) (2015)
- [Exploring the Function Space of Deep-Learning Machines](https://mlsanity.com/paper/1708.01422/) (2017)
- [Self-Organized Criticality in the Brain](https://mlsanity.com/paper/2102.09124/) (2021)
- [Statistical mechanics of the US Supreme Court](https://mlsanity.com/paper/1306.5004/) (2013)
- [Learning with hidden variables](https://mlsanity.com/paper/1506.00354/) (2015)
- [Optimal encoding in stochastic latent-variable Models](https://mlsanity.com/paper/1802.10361/) (2018)
- [Minimum Description Length codes are critical](https://mlsanity.com/paper/1809.00652/) (2018)

## Built on by (most cited first, 2 of 2)

- [Quantifying Relevance in Learning and Inference](https://mlsanity.com/paper/2202.00339/) (2022)
- [Absolute abstraction: a renormalisation group approach](https://mlsanity.com/paper/2407.01656/) (2024)

## Abstract (arXiv)

We study a generic ensemble of deep belief networks which is parametrized by the distribution of energy levels of the hidden states of each layer. We show that, within a random energy approach, statistical dependence can propagate from the visible to deep layers only if each layer is tuned close to the critical point during learning. As a consequence, efficiently trained learning machines are characterised by a broad distribution of energy levels. The analysis of Deep Belief Networks and Restricted Boltzmann Machines on different datasets confirms these conclusions.