mlsanityData to 7 Oct 2026

Entropy, concentration, and learning: a statistical mechanics primer

arXiv 2409.18630 ↗First posted 27 Sep 2024cs.LG cond-mat.stat-mech cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence models trained through loss minimization have demonstrated significant success, grounded in principles from fields like information theory and statistical physics. This work explores these established connections through the lens of statistical mechanics, starting from first-principles sample concentration behaviors that underpin AI and machine learning. Our development of statistical mechanics for modeling highlights the key role of exponential families, and quantities of statistics, physics, and information theory.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 101 references that are in the map. First posted on arXiv 27 Sep 2024.

Builds on

19 of 101 references.

References · 19

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2019Invariant Risk Minimizationcited by 1,832
  4. 2016Variational Inference: A Review for Statisticianscited by 1,466
  5. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  6. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,232
  7. 2019An Introduction to Variational Autoencoderscited by 997
  8. 2019Agnostic Federated Learningcited by 451
  9. 2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
  10. 2013A survey of the Schr"odinger problem and some of its connections with optimal transportcited by 286
  11. 2023Diffusion Schr"odinger Bridge Matchingcited by 151
  12. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
7 more
  1. 2017Inverse statistical problems: from the inverse Ising problem to data sciencecited by 36
  2. 2022Bregman Deviations of Generic Exponential Familiescited by 13
  3. 2015Optimally Combining Classifiers Using Unlabeled Datacited by 9
  4. 2015Optimal Binary Classifier Aggregation for General Lossescited by 6
  5. 2020Sharp finite-sample concentration of independent variablescited by 3
  6. 2023Exact minimax entropy models of large-scale neuronal activitycited by 3
  7. 2016Optimal Binary Autoencoding with Pairwise Correlationscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Laws of thermodynamics for exponential familiescited by 0