mlsanityData to 14 Oct 2026

Deep Transfer Learning: Model Framework and Error Analysis

arXiv 2410.09383 ↗First posted 12 Oct 2024cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper presents a framework for deep transfer learning, which aims to leverage information from multi-domain upstream data with a large number of samples n to a single-domain downstream task with a considerably smaller number of samples m, where m ll n, in order to enhance performance on downstream task. Our framework offers several intriguing features. First, it allows the existence of both shared and domain-specific features across multi-domain data and provides a framework for automatic identification, achieving precise transfer and utilization of information. Second, the framework explicitly identifies upstream features that contribute to downstream tasks, establishing clear relationships between upstream domains and downstream tasks, thereby enhancing interpretability. Error analysis shows that our framework can significantly improve the convergence rate for learning Lipschitz functions in downstream supervised tasks, reducing it from tildeO(m^-frac12(d+2)+n^-frac12(d+2)) ("no transfer") to tildeO(m^-frac12(d^*+3) + n^-frac12(d+2)) ("partial transfer"), and even to tildeO(m-1/2+n^-frac12(d+2)) ("complete transfer"), where d^* ll d and d is the dimension of the observed data. Our theoretical findings are supported by empirical experiments on image classification and regression datasets.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 42 of its 83 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

42 of 83 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,044
  5. 2019Invariant Risk Minimizationcited by 1,854
  6. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,188
  7. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  8. 2020In Search of Lost Domain Generalizationcited by 1,054
  9. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  10. 2017Deeper, Broader and Artier Domain Generalizationcited by 986
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  12. 2018Recognition in Terra Incognitacited by 658
30 more
  1. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
  2. 2019Bridging Theory and Algorithm for Domain Adaptationcited by 332
  3. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 311
  4. 2020A Brief Review of Domain Adaptationcited by 288
  5. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 233
  6. 2020Few-Shot Learning via Learning the Representation, Provablycited by 214
  7. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  8. 2020Provable Meta-Learning of Linear Representationscited by 161
  9. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146
  10. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  11. 2020Efficient Domain Generalization via Common-Specific Low-Rank Decompositioncited by 134
  12. 2020Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimalitycited by 90
  13. 2019Support and Invertibility in Domain-Invariant Representationscited by 87
  14. 2019Deep ReLU network approximation of functions on a manifoldcited by 74
  15. 2019Nonparametric Regression on Low-Dimensional Manifolds using Deep ReLU Networks : Function Approximation and Statistical Recoverycited by 73
  16. 2018Marginal Singularity, and the Benefits of Labels in Covariate-Shiftcited by 65
  17. 2019Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifiercited by 65
  18. 2021An error analysis of generative adversarial networks for learning distributionscited by 36
  19. 2022Cross-Modal Adapter for Vision-Language Retrievalcited by 35
  20. 2022Factor Augmented Sparse Throughput Deep ReLU Neural Networks for High Dimensional Regressioncited by 31
  21. 2022Improving Multi-Task Generalization via Regularizing Spurious Correlationcited by 28
  22. 2021Weighted Training for Cross-Task Learningcited by 22
  23. 2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 22
  24. 2024Transfer Learning for Nonparametric Regression: Non-asymptotic Minimax Analysis and Adaptive Procedurecited by 22
  25. 2020A Sample Complexity Separation between Non-Convex and Convex Meta-Learningcited by 17
  26. 2020Deep Dimension Reduction for Supervised Representation Learningcited by 15
  27. 2021On Learning Domain-Invariant Representations for Transfer Learning with Multiple Sourcescited by 15
  28. 2023Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimensioncited by 15
  29. 2014Fast Computing for Distance Covariancecited by 14
  30. 2023Prominent Roles of Conditionally Invariant Components in Domain Adaptation: Theory and Algorithmscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Transfer Learning through Enhanced Sufficient Representation: Enriching Source Domain Knowledge with Target Datacited by 1
  2. 2026Enhancing Spectral Embedding through Robust and Flexible Knowledge Transfer in Electronic Health Recordscited by 0
  3. 2026Adaptive deep nonparametric regression from dependent data under covariate shiftcited by 0
  4. 2026Transfer Learning in Nonparametric Regression with Deep ReLU Networkscited by 0
  5. 2026Sufficiently Reduced Distributional Regressioncited by 0