mlsanityData to 7 Oct 2026

What is being transferred in transfer learning?

arXiv 2008.11687 ↗First posted 26 Aug 2020cs.LG stat.MLcited by 365

Not on the map. Why?

Abstract (arXiv)

One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce. Despite ample adaptation of transfer learning in various deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analyses to address these fundamental questions. Through a series of analyses on transferring to block-shuffled images, we separate the effect of feature reuse from learning low-level statistics of data and show that some benefit of transfer learning comes from the latter. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.

Key facts. Cited by 365 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 45; 2022: 82; 2023: 59; 2024: 87; 2025: 65; 2026: 27.

365 papers.

Sort:

By area:

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2022Editing Models with Task Arithmeticcited by 1,076
  3. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  7. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  8. 2021Big Self-Supervised Models Advance Medical Image Classificationcited by 320
  9. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  10. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  11. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  12. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
38 more
  1. 2021BENDR: using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG datacited by 194
  2. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  3. 2022Patching open-vocabulary models by interpolating weightscited by 187
  4. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  5. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  6. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 155
  7. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  8. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  9. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  10. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  11. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  12. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  13. 2022FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learningcited by 120
  14. 2024Model merging with SVD to tie the Knotscited by 118
  15. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  16. 2022Fusing finetuned models for better pretrainingcited by 110
  17. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 105
  18. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  19. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  20. 2021Representational Continuity for Unsupervised Continual Learningcited by 95
  21. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  22. 2023Deep Model Fusion: A Surveycited by 89
  23. 2021Continual Learning in the Teacher-Student Setup: Impact of Task Similaritycited by 86
  24. 2022Deep transfer operator learning for partial differential equations under conditional shiftcited by 83
  25. 2022Reliability of CKA as a Similarity Measure in Deep Learningcited by 82
  26. 2023Modular Deep Learningcited by 82
  27. 2022When Do Flat Minima Optimizers Work?cited by 78
  28. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  29. 2023A Foundation Language-Image Model of the Retina (FLAIR): Encoding Expert Knowledge in Text Supervisioncited by 78
  30. 2021Similarity and Matching of Neural Network Representationscited by 77
  31. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  32. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  33. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  34. 2021Transferable Visual Words: Exploiting the Semantics of Anatomical Patterns for Self-supervised Learningcited by 68
  35. 2022What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factorscited by 61
  36. 2021Randomness In Neural Network Training: Characterizing The Impact of Toolingcited by 57
  37. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 57
  38. 2022Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representationscited by 57

50 most cited of 365 All 365 (JSON)