A statistical physics framework for optimal learning
Not on the map. Why?
Abstract (arXiv)
Learning is a complex dynamical process shaped by a range of interconnected decisions. Careful design of hyperparameter schedules for artificial neural networks or efficient allocation of cognitive resources by biological learners can dramatically affect performance. Yet, theoretical understanding of optimal learning strategies remains sparse, especially due to the intricate interplay between evolving metaparameters and nonlinear learning dynamics. The search for optimal protocols is further hindered by the high dimensionality of the learning space, often resulting in predominantly heuristic, difficult to interpret, and computationally demanding solutions. Here, we combine statistical physics with control theory in a unified theoretical framework to identify optimal learning protocols in prototypical neural network models. In the high-dimensional limit, we derive closed-form ordinary differential equations that track online stochastic gradient descent through low-dimensional order parameters. We formulate the design of learning protocols as an optimal control problem directly on the dynamics of the order parameters with the goal of minimizing the generalization error. This formulation encompasses a variety of learning scenarios, optimization constraints, and control budgets. We apply it to representative cases, including optimal curricula, adaptive dropout regularization and noise schedules in denoising autoencoders. We find nontrivial yet interpretable strategies highlighting how optimal protocols mediate learning trade-offs. Our results establish a principled foundation for understanding and designing optimal protocols and suggest a path toward a theory of meta-learning grounded in statistical physics.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 78 of its 134 references that are in the map. First posted on arXiv 10 Jul 2025.
Builds on
78 of 134 references.
References · 78
- 2017Attention Is All You Needcited by 60,361
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2017Squeeze-and-Excitation Networkscited by 4,789
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2019Are Sixteen Heads Really Better than One?cited by 816
- 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
- 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 471
- 2021Dynamic Neural Networks: A Surveycited by 408
38 more
- 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
- 2015Gradient-based Hyperparameter Optimization through Reversible Learningcited by 360
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
- 2019On The Power of Curriculum Learning in Training Deep Networkscited by 286
- 2019Adaptively Sparse Transformerscited by 155
- 2019Adaptive Attention Span in Transformerscited by 149
- 2020Generalisation error in learning with random features and the hidden manifold modelcited by 127
- 2021Learning curves of generic features maps for realistic datasets with a teacher-student modelcited by 126
- 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
- 2017Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Modelscited by 116
- 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
- 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
- 2020When Do Curricula Work?cited by 106
- 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 105
- 2020Neural Pruning via Growing Regularizationcited by 97
- 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 88
- 2022How catastrophic can catastrophic forgetting be in linear regression?cited by 81
- 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
- 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
- 2014Curriculum Learning of Multiple Taskscited by 66
- 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
- 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
- 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 52
- 2017Curriculum Dropoutcited by 50
- 2024Why Warmup the Learning Rate? Underlying Mechanisms and Improvementscited by 46
- 2023Bayes-optimal Learning of Deep Random Networks of Extensive-widthcited by 41
- 2023Analysis of learning a flow-based generative model from limited sample complexitycited by 38
- 2022The Neural Race Reduction: Dynamics of Abstraction in Gated Networkscited by 37
- 2023Continual Learning in Linear Classification on Separable Datacited by 34
- 2018Theory of Curriculum Learning, with Convex Loss Functionscited by 33
- 2016Coupling Adaptive Batch Sizes with Learning Ratescited by 30
- 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
- 2017AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networkscited by 29
- 2021An Analytical Theory of Curriculum Learning in Teacher-Student Networkscited by 27
- 2022The dynamics of representation learning in shallow, non-linear autoencoderscited by 27
- 2019Gated Linear Networkscited by 26
- 2025Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networkscited by 25
- 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
50 most cited of 78 All 78 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.