# A Free-Energy Principle for Representation Learning

arXiv: https://arxiv.org/abs/2002.12406 · HTML: https://mlsanity.com/paper/2002.12406/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2020.

## Built on by (most cited first, 7 of 7)

- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [Controllable Guarantees for Fair Outcomes via Contrastive Information Estimation](https://mlsanity.com/paper/2101.04108/) (2021)
- [A picture of the space of typical learnable tasks](https://mlsanity.com/paper/2210.17011/) (2022)
- [SGD as Free Energy Minimization: A Thermodynamic View on Neural Network Training](https://mlsanity.com/paper/2505.23489/) (2025)
- [Rethinking Neural Networks With Benford's Law](https://mlsanity.com/paper/2102.03313/) (2021)
- [Deep Reference Priors: What is the best way to pretrain a model?](https://mlsanity.com/paper/2202.00187/) (2022)
- [Can Stationary Distributions of Scale-Invariant Neural Networks Be Described by the Thermodynamics of an Ideal Gas?](https://mlsanity.com/paper/2511.07308/) (2025)

## Abstract (arXiv)

This paper employs a formal connection of machine learning with thermodynamics to characterize the quality of learnt representations for transfer learning. We discuss how information-theoretic functional such as rate, distortion and classification loss of a model lie on a convex, so-called equilibrium surface.We prescribe dynamical processes to traverse this surface under constraints, e.g., an iso-classification process that trades off rate and distortion to keep the classification loss unchanged. We demonstrate how this process can be used for transferring representations from a source dataset to a target dataset while keeping the classification loss constant. Experimental validation of the theoretical results is provided on standard image-classification datasets.