mlsanityData to 7 Oct 2026

Stochastic Thermodynamics of Learning Parametric Probabilistic Models

arXiv 2310.19802 ↗First posted 4 Oct 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We have formulated a family of machine learning problems as the time evolution of Parametric Probabilistic Models (PPMs), inherently rendering a thermodynamic process. Our primary motivation is to leverage the rich toolbox of thermodynamics of information to assess the information-theoretic content of learning a probabilistic model. We first introduce two information-theoretic metrics: Memorized-information (M-info) and Learned-information (L-info), which trace the flow of information during the learning process of PPMs. Then, we demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 17 of its 43 references that are in the map. First posted on arXiv 4 Oct 2023.

Builds on

17 of 43 references.

References · 17

  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  4. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  5. 2021How to Train Your Energy-Based Modelscited by 271
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  7. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  8. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  9. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  10. 2019Where is the Information in a Deep Neural Network?cited by 40
  11. 2020On Information Plane Analyses of Neural Network Classifiers -- A Reviewcited by 35
  12. 2019How noise affects the Hessian spectrum in overparameterized neural networkscited by 17
5 more
  1. 2022An Information-Theoretic Framework for Supervised Learningcited by 6
  2. 2023Noise Balance and Stationary Distribution of Stochastic Gradient Descentcited by 6
  3. 2023Machine learning in and out of equilibriumcited by 5
  4. 2022Mutual Information Learned Classifiers: an Information-theoretic Viewpoint of Training Deep Learning Classification Systemscited by 2
  5. 2023Anti-Correlated Noise in Epoch-Based Stochastic Gradient Descent: Implications for Weight Variances in Flat Directionscited by 2

Built on by

Not cited yet.