Learning Dynamics of Meta-Learning in Small Model Pretraining
Not on the map. Why?
Abstract (arXiv)
Large language models are powerful but costly. We ask whether meta-learning can make the pretraining of small language models not only better but also more interpretable. We integrate first-order MAML with subset-masked LM pretraining, producing four LLama-style decoder-only models (11M-570M params), and evaluate it on a fundamental NLP task with many settings and real-world applications. Compared with vanilla training, our model (i) reaches the same loss up to 1.6x sooner, (ii) improves F1 on multilingual Universal NER under equal compute, and (iii) makes the training dynamics easy to read: first the network's representations fan out ("diversify") and later they collapse into a smaller, shared subspace ("compress"). This two-stage shift shows up as a rise-and-fall in both effective-rank curves and attention-head entropy. The same curves pinpoint which layers specialise earliest and which later reconverge, giving a compact, interpretable signature of meta-adaptation. Code, checkpoints and WandB logs are released.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 60 references that are in the map. First posted on arXiv 4 Aug 2025.
Builds on
30 of 60 references.
References · 30
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2020GLU Variants Improve Transformercited by 1,804
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2018On First-Order Meta-Learning Algorithmscited by 1,018
- 2022In-context Learning and Induction Headscited by 964
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
18 more
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
- 2019Meta-Learning with Implicit Gradientscited by 525
- 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
- 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
- 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
- 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
- 2021The Low-Rank Simplicity Bias in Deep Networkscited by 154
- 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 54
- 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
- 2021Meta-Learning to Improve Pre-Trainingcited by 27
- 2023Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmarkcited by 26
- 2020A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Textscited by 21
- 2024Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneckcited by 21
- 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 19
- 2022Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptationcited by 14
- 2022E-NER -- An Annotated Named Entity Recognition Corpus of Legal Textcited by 13
- 2023Is Pre-training Truly Better Than Meta-Learning?cited by 6
Built on by
Citing papers per year: 2025: 2; 2026: 1.
3 papers.