Transfer Learning in Infinite Width Feature Learning Networks
Not on the map. Why?
Abstract (arXiv)
We develop a theory of transfer learning in infinitely wide neural networks under gradient flow that quantifies when pretraining on a source task improves generalization on a target task. We analyze both (i) fine-tuning, when the downstream predictor is trained on top of source-induced features and (ii) a jointly rich setting, where both pretraining and downstream tasks can operate in a feature learning regime, but the downstream model is initialized with the features obtained after pre-training. In this setup, the summary statistics of randomly initialized networks after a rich pre-training are adaptive kernels which depend on both source data and labels. For (i), we analyze the performance of a readout for different pretraining data regimes. For (ii), the summary statistics after learning the target task are still adaptive kernels with features from both source and target tasks. We test our theory on linear and polynomial regression tasks as well as real datasets. Our theory allows interpretable conclusions on performance, which depend on the amount of data on both tasks, the alignment between tasks, and the feature learning strength.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 61 of its 91 references that are in the map. First posted on arXiv 6 Jul 2025.
Builds on
61 of 91 references.
References · 61
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018On the Spectral Bias of Neural Networkscited by 1,321
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2017Deep Neural Networks as Gaussian Processescited by 617
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
38 more
- 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 508
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2018A Survey on Deep Transfer Learningcited by 417
- 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
- 2017High-dimensional dynamics of generalization error in neural networkscited by 238
- 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 229
- 2021Revisiting Model Stitching to Compare Neural Representationscited by 203
- 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
- 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
- 2020Feature Learning in Infinite-Width Neural Networkscited by 182
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
- 2021The Principles of Deep Learning Theorycited by 169
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
- 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
- 2024A Dynamical Model of Neural Scaling Lawscited by 105
- 2022Self-Consistent Dynamical Field Theory of Kernel Evolution in Wide Neural Networkscited by 104
- 2023Grokking as the Transition from Lazy to Rich Training Dynamicscited by 102
- 2020Statistical Mechanics of Deep Linear Neural Networks: The Back-Propagating Kernel Renormalizationcited by 71
- 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
- 2021The staircase property: How hierarchical structure can guide deep learningcited by 62
- 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
- 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
- 2021Separation of Scales and a Thermodynamic Description of Feature Learning in Some CNNscited by 53
- 2023Grokking as a First Order Phase Transition in Two Layer Networkscited by 48
- 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
- 2019Why bigger is not always better: on finite and infinite neural networkscited by 41
- 2016Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemescited by 40
- 2015Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapsescited by 39
- 2021Unveiling the structure of wide flat minima in neural networkscited by 27
- 2021The effective noise of Stochastic Gradient Descentcited by 25
- 2018Marvels and Pitfalls of the Langevin Algorithm in Noisy High-dimensional Inferencecited by 24
- 2021Probing transfer learning with a model of synthetic correlated datasetscited by 24
- 2020For self-supervised learning, Rationality implies generalization, provablycited by 22
- 2021Phase Transitions in Transfer Learning for High-Dimensional Perceptronscited by 21
- 2021Out-of-Distribution Generalization in Kernel Regressioncited by 21
- 2021Learning through atypical "phase transitions" in overparameterized neural networkscited by 21
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.