# Entropy, concentration, and learning: a statistical mechanics primer

arXiv: https://arxiv.org/abs/2409.18630 · HTML: https://mlsanity.com/paper/2409.18630/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 101 references that are in the map.
- First posted on arXiv 27 Sep 2024.

## Builds on (most cited first, 19 of 19)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [An Introduction to Variational Autoencoders](https://mlsanity.com/paper/1906.02691/) (2019)
- [Agnostic Federated Learning](https://mlsanity.com/paper/1902.00146/) (2019)
- [Learning Models with Uniform Performance via Distributionally Robust Optimization](https://mlsanity.com/paper/1810.08750/) (2018)
- [A survey of the Schr"odinger problem and some of its connections with optimal transport](https://mlsanity.com/paper/1308.0215/) (2013)
- [Diffusion Schr"odinger Bridge Matching](https://mlsanity.com/paper/2303.16852/) (2023)
- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [Inverse statistical problems: from the inverse Ising problem to data science](https://mlsanity.com/paper/1702.01522/) (2017)
- [Bregman Deviations of Generic Exponential Families](https://mlsanity.com/paper/2201.07306/) (2022)
- [Optimally Combining Classifiers Using Unlabeled Data](https://mlsanity.com/paper/1503.01811/) (2015)
- [Optimal Binary Classifier Aggregation for General Losses](https://mlsanity.com/paper/1510.00452/) (2015)
- [Sharp finite-sample concentration of independent variables](https://mlsanity.com/paper/2008.13293/) (2020)
- [Exact minimax entropy models of large-scale neuronal activity](https://mlsanity.com/paper/2402.00007/) (2023)
- [Optimal Binary Autoencoding with Pairwise Correlations](https://mlsanity.com/paper/1611.02268/) (2016)

## Built on by (most cited first, 2 of 2)

- [Information from coincidences](https://mlsanity.com/paper/2606.25042/) (2026)
- [Laws of thermodynamics for exponential families](https://mlsanity.com/paper/2501.02071/) (2025)

## Abstract (arXiv)

Artificial intelligence models trained through loss minimization have demonstrated significant success, grounded in principles from fields like information theory and statistical physics. This work explores these established connections through the lens of statistical mechanics, starting from first-principles sample concentration behaviors that underpin AI and machine learning. Our development of statistical mechanics for modeling highlights the key role of exponential families, and quantities of statistics, physics, and information theory.