Early Period of Training Impacts Adaptation for Out-of-Distribution Generalization: An Empirical Study
Not on the map. Why?
Abstract (arXiv)
Prior research shows that differences in the early period of neural network training significantly impact the performance of in-distribution (ID) data of tasks. Yet, the implications of early learning dynamics on out-of-distribution (OOD) generalization remain poorly understood, primarily due to the complexities and limitations of existing analytical techniques. In this work, we investigate the relationship between learning dynamics, OOD generalization under covariate shift and the early period of neural network training. We utilize the trace of Fisher Information and sharpness, focusing on gradual unfreezing (i.e., progressively unfreezing parameters during training) as our methodology for investigation. Through a series of empirical experiments, we show that 1) changing the number of trainable parameters during the early period of training via gradual unfreezing can significantly improve OOD results; 2) the trace of Fisher Information and sharpness can be used as indicators for the removal of gradual unfreezing during the early period of training for better OOD generalization. Our experiments on both image and text data show that the early period of training is a general phenomenon that can provide Pareto improvements in ID and OOD performance with minimal complexity. Our work represents a first step towards understanding how early learning dynamics affect neural network OOD generalization under covariate shift and suggests a new avenue to improve and study this problem.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 59 of its 74 references that are in the map. First posted on arXiv 22 Mar 2024.
Builds on
59 of 74 references.
References · 59
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
38 more
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
- 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
- 2020In Search of Lost Domain Generalizationcited by 1,042
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
- 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
- 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
- 2017Exploring Generalization in Deep Learningcited by 595
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
- 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
- 2019Fantastic Generalization Measures and Where to Find Themcited by 507
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
- 2020AdapterHub: A Framework for Adapting Transformerscited by 441
- 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
- 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
- 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
- 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
- 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
- 2017Three Factors Influencing Minima in SGDcited by 270
- 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
- 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
- 2021Training Neural Networks with Fixed Sparse Maskscited by 212
- 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
- 2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
- 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
- 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
- 2019MNIST-C: A Robustness Benchmark for Computer Visioncited by 137
- 2021TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learningcited by 116
- 2020The Early Phase of Neural Network Trainingcited by 114
- 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
- 2022Wild-Time: A Benchmark of in-the-Wild Distribution Shift over Timecited by 85
- 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
- 2022GradMax: Growing Neural Networks using Gradient Informationcited by 64
- 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
- 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
- 2018MTNT: A Testbed for Machine Translation of Noisy Textcited by 54
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.