mlsanityData to 14 Oct 2026

Understanding Knowledge Transferability for Transfer Learning: A Survey

arXiv 2507.03175 ↗First posted 3 Jul 2025cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Transfer learning has become an essential paradigm in artificial intelligence, enabling the transfer of knowledge from a source task to improve performance on a target task. This approach, particularly through techniques such as pretraining and fine-tuning, has seen significant success in fields like computer vision and natural language processing. However, despite its widespread use, how to reliably assess the transferability of knowledge remains a challenge. Understanding the theoretical underpinnings of each transferability metric is critical for ensuring the success of transfer learning. In this survey, we provide a unified taxonomy of transferability metrics, categorizing them based on transferable knowledge types and measurement granularity. This work examines the various metrics developed to evaluate the potential of source knowledge for transfer learning and their applicability across different learning paradigms emphasizing the need for careful selection of these metrics. By offering insights into how different metrics work under varying conditions, this survey aims to guide researchers and practitioners in selecting the most appropriate metric for specific applications, contributing to more efficient, reliable, and trustworthy AI systems. Finally, we discuss some open challenges in this field and propose future research directions to further advance the application of transferability metrics in trustworthy transfer learning.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 58 of its 99 references that are in the map. First posted on arXiv 3 Jul 2025.

Builds on

58 of 99 references.

References · 58

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  6. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  7. 2016Learning without Forgettingcited by 2,496
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2015Learning Transferable Features with Deep Adaptation Networkscited by 1,329
  11. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
  12. 2017Adversarial Discriminative Domain Adaptationcited by 1,311
38 more
  1. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  3. 2014How transferable are features in deep neural networks?cited by 1,225
  4. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  5. 2021Generalized Out-of-Distribution Detection: A Surveycited by 802
  6. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  7. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  8. 2016Deep Transfer Learning with Joint Adaptation Networkscited by 612
  9. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 372
  10. 2016Kernel Mean Embedding of Distributions: A Review and Beyondcited by 364
  11. 2016Unsupervised Domain Adaptation with Residual Transfer Networkscited by 341
  12. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  13. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  14. 2019Task2Vec: Task Embedding for Meta-Learningcited by 205
  15. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 188
  16. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  17. 2020Geometric Dataset Distances via Optimal Transportcited by 158
  18. 2016Correlation Alignment for Unsupervised Domain Adaptationcited by 147
  19. 2019Transferability and Hardness of Supervised Classification Taskscited by 127
  20. 2018Adapting Auxiliary Losses Using Gradient Similaritycited by 100
  21. 2021MOOD: Multi-level Out-of-distribution Detectioncited by 94
  22. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  23. 2022An Information-Theoretic Approach to Transferability in Task Transfer Learningcited by 81
  24. 2020Active Domain Adaptation via Clustering Uncertainty-weighted Embeddingscited by 72
  25. 2019An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Modelscited by 39
  26. 2023OT-Attack: Enhancing Adversarial Transferability of Vision-Language Models via Optimal Transport Optimizationcited by 37
  27. 2021Quantifying and Improving Transferability in Domain Generalizationcited by 35
  28. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
  29. 2022How stable are Transferability Metrics evaluations?cited by 30
  30. 2019Deep Model Transferability from Attribution Mapscited by 28
  31. 2021Transferability Metrics for Selecting Source Model Ensemblescited by 24
  32. 2019Meta-Transfer Learning through Hard Taskscited by 23
  33. 2020Duality Diagram Similarity: a generic framework for initialization selection in task transfer learningcited by 23
  34. 2020Auxiliary Task Reweighting for Minimum-data Learningcited by 23
  35. 2021Domain-Adversarial Training of Self-Attention Based Networks for Land Cover Classification using Multi-temporal Sentinel-2 Satellite Imagerycited by 18
  36. 2022Transferability-Guided Cross-Domain Cross-Task Transfer Learningcited by 14
  37. 2024Which Model to Transfer? A Survey on Transferability Estimationcited by 13
  38. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Monotonicity as an Architectural Bias for Robust Language Modelscited by 0
  2. 2026Adversarial Domain Adaptation Enables Knowledge Transfer Across Heterogeneous RNA-Seq Datasetscited by 0
  3. 2026Heterogeneity-Aware Dataset Scheduling for Efficient Audio Large Language Model Trainingcited by 0
  4. 2026RADAR: Relative Angular Divergence Across Representationscited by 0