mlsanityData to 7 Oct 2026

Dynamics of Meta-learning Representation in the Teacher-student Scenario

arXiv 2408.12545 ↗First posted 22 Aug 2024cs.LG cond-mat.dis-nncited by 0

Not on the map. Why?

Abstract (arXiv)

Gradient-based meta-learning algorithms have gained popularity for their ability to train models on new tasks using limited data. Empirical observations indicate that such algorithms are able to learn a shared representation across tasks, which is regarded as a key factor in their success. However, the in-depth theoretical understanding of the learning dynamics and the origin of the shared representation remains underdeveloped. In this work, we investigate the meta-learning dynamics of nonlinear two-layer neural networks trained on streaming tasks in the teacher-student scenario. Through the lens of statistical physics analysis, we characterize the macroscopic behavior of the meta-training processes, the formation of the shared representation, and the generalization ability of the model on new tasks. The analysis also points to the importance of the choice of certain hyperparameters of the learning algorithms.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 55 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

27 of 55 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  10. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  11. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  12. 2016Learning to learn by gradient descent by gradient descentcited by 616
15 more
  1. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  2. 2019Meta-Learning with Implicit Gradientscited by 523
  3. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  4. 2017Meta-SGD: Learning to Learn Quickly for Few-Shot Learningcited by 377
  5. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  6. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  7. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  8. 2017The loss surface of deep and wide neural networkscited by 74
  9. 2022MAML and ANIL Provably Learn Representationscited by 30
  10. 2021Statistical Mechanical Analysis of Catastrophic Forgetting in Continual Learning with Teacher and Student Networkscited by 22
  11. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 13
  12. 2022Provable Generalization of Overparameterized Meta-learning Trained with SGDcited by 9
  13. 2022Statistical mechanics of continual learning: variational principle and mean-field potentialcited by 9
  14. 2017Exploring the Function Space of Deep-Learning Machinescited by 7
  15. 2020Space of Functions Computed by Deep-Layered Machinescited by 2

Built on by

Not cited yet.