mlsanityData to 7 Oct 2026

Probing transfer learning with a model of synthetic correlated datasets

arXiv 2106.05418 ↗First posted 9 Jun 2021cs.LG cond-mat.dis-nncited by 24

Not on the map. Why?

Abstract (arXiv)

Transfer learning can significantly improve the sample efficiency of neural networks, by exploiting the relatedness between a data-scarce target task and a data-abundant source task. Despite years of successful applications, transfer learning practice often relies on ad-hoc solutions, while theoretical understanding of these procedures is still limited. In the present work, we re-think a solvable model of synthetic data as a framework for modeling correlation between data-sets. This setup allows for an analytic characterization of the generalization performance obtained when transferring the learned feature map from the source to the target task. Focusing on the problem of training two-layer networks in a binary classification setting, we show that our model can capture a range of salient features of transfer learning with real data. Moreover, by exploiting parametric control over the correlation between the two data-sets, we systematically investigate under which conditions the transfer of features is beneficial for generalization.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 39 of its 76 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

39 of 76 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  6. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  7. 2014How transferable are features in deep neural networks?cited by 1,216
  8. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  9. 2017A Survey on Multi-Task Learningcited by 838
  10. 2018Do Better ImageNet Models Transfer Better?cited by 560
  11. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 410
  12. 2019Transfusion: Understanding Transfer Learning for Medical Imagingcited by 366
27 more
  1. 2020What is being transferred in transfer learning?cited by 365
  2. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  3. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  4. 2018GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classificationcited by 242
  5. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  6. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  7. 2016Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learningcited by 209
  8. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  9. 2020Geometric Dataset Distances via Optimal Transportcited by 157
  10. 2018Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networkscited by 146
  11. 2019Data augmentation using learned transformations for one-shot medical image segmentationcited by 138
  12. 2020Generalisation error in learning with random features and the hidden manifold modelcited by 126
  13. 2019Transferability and Hardness of Supervised Classification Taskscited by 125
  14. 2020Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regimecited by 95
  15. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  16. 2018A jamming transition from under- to over-parametrization affects loss landscape and generalizationcited by 65
  17. 2019Transfer Adaptation Learning: A Decade Surveycited by 52
  18. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 52
  19. 2018The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regressioncited by 45
  20. 2019The Information Complexity of Learning Tasks, their Structure and their Distancecited by 35
  21. 2021Phase Transitions in Transfer Learning for High-Dimensional Perceptronscited by 21
  22. 2019Asymptotic learning curves of kernel methods: empirical data v.s. Teacher-Student paradigmcited by 20
  23. 2020Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Taskscited by 15
  24. 2020An Information-Geometric Distance on the Space of Taskscited by 15
  25. 2021Similarity of Classification Taskscited by 7
  26. 2016Towards Automated Melanoma Screening: Exploring Transfer Learning Schemescited by 2
  27. 2017DeepRadiologyNet: Radiologist Level Pathology Detection in CT Head Imagescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 3; 2024: 7; 2025: 5; 2026: 3.

24 papers.

Sort:

  1. 2021A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learningcited by 54
  2. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  3. 2022Gaussian Universality of Perceptrons with Random Labelscited by 26
  4. 2022Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activationcited by 16
  5. 2024Optimal Protocols for Continual Learning via Statistical Physics and Control Theorycited by 14
  6. 2024Order parameters and phase transitions of continual learning in deep neural networkscited by 13
  7. 2024Features are fate: a theory of transfer learning in high-dimensional regressioncited by 10
  8. 2024On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theorycited by 10
  9. 2022Bias-inducing geometries: an exactly solvable data model with fairness implicationscited by 8
  10. 2025Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Modelscited by 6
  11. 2023The RL Perceptron: Generalisation Dynamics of Policy Learning in High Dimensionscited by 5
  12. 2023Generalization Performance of Transfer Learning: Overparameterized and Underparameterized Regimescited by 4
12 more
  1. 2024TL-PCA: Transfer Learning of Principal Component Analysiscited by 4
  2. 2026Positive Distribution Shift as a Framework for Understanding Tractable Learningcited by 3
  3. 2025Transfer Learning in Infinite Width Feature Learning Networkscited by 2
  4. 2025The Effect of Label Noise on the Information Content of Neural Representationscited by 2
  5. 2025Beyond Real Data: Synthetic Data through the Lens of Regularizationcited by 2
  6. 2022The impact of memory on learning sequence-to-sequence taskscited by 1
  7. 2022Frozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networkscited by 1
  8. 2025Cross-functional transferability in universal machine learning interatomic potentialscited by 1
  9. 2023Optimal transfer protocol by incremental layer defrostingcited by 0
  10. 2024The twin peaks of learning neural networkscited by 0
  11. 2026Transfer Learning of Linear Regression with Multiple Pretrained Models: Benefiting from More Pretrained Models via Overparameterization Debiasingcited by 0
  12. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 0