# Nonequilibrium thermodynamics of self-supervised learning

arXiv: https://arxiv.org/abs/2106.08981 · HTML: https://mlsanity.com/paper/2106.08981/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 32 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Builds on (most cited first, 5 of 5)

- [Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models](https://mlsanity.com/paper/1708.03395/) (2017)
- [Deep autoregressive models for the efficient variational simulation of many-body quantum systems](https://mlsanity.com/paper/1902.04057/) (2019)
- [Inverse statistical problems: from the inverse Ising problem to data science](https://mlsanity.com/paper/1702.01522/) (2017)
- [Learning physical properties of liquid crystals with deep convolutional neural networks](https://mlsanity.com/paper/2004.01691/) (2020)
- [Estimating physical properties from liquid crystal textures via machine learning and complexity-entropy methods](https://mlsanity.com/paper/1901.01754/) (2019)

## Abstract (arXiv)

Self-supervised learning (SSL) of energy based models has an intuitive relation to equilibrium thermodynamics because the softmax layer, mapping energies to probabilities, is a Gibbs distribution. However, in what way SSL is a thermodynamic process? We show that some SSL paradigms behave as a thermodynamic composite system formed by representations and self-labels in contact with a nonequilibrium reservoir. Moreover, this system is subjected to usual thermodynamic cycles, such as adiabatic expansion and isochoric heating, resulting in a generalized Gibbs ensemble (GGE). In this picture, we show that learning is seen as a demon that operates in cycles using feedback measurements to extract negative work from the system. As applications, we examine some SSL algorithms using this idea.