Transferability Estimation Based On Principal Gradient Expectation
Not on the map. Why?
Abstract (arXiv)
Transfer learning aims to improve the performance of target tasks by transferring knowledge acquired in source tasks. The standard approach is pre-training followed by fine-tuning or linear probing. Especially, selecting a proper source domain for a specific target domain under predefined tasks is crucial for improving efficiency and effectiveness. It is conventional to solve this problem via estimating transferability. However, existing methods can not reach a trade-off between performance and cost. To comprehensively evaluate estimation methods, we summarize three properties: stability, reliability and efficiency. Building upon them, we propose Principal Gradient Expectation(PGE), a simple yet effective method for assessing transferability. Specifically, we calculate the gradient over each weight unit multiple times with a restart scheme, and then we compute the expectation of all gradients. Finally, the transferability between the source and target is estimated by computing the gap of normalized principal gradients. Extensive experiments show that the proposed metric is superior to state-of-the-art methods on all properties.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 52 of its 77 references that are in the map. First posted on arXiv 29 Nov 2022.
Builds on
52 of 77 references.
References · 52
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
38 more
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
- 2018Deep contextualized word representationscited by 2,285
- 2016Matching Networks for One Shot Learningcited by 2,246
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
- 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
- 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
- 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
- 2019A Comprehensive Survey on Transfer Learningcited by 1,154
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
- 2018MnasNet: Platform-Aware Neural Architecture Search for Mobilecited by 937
- 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
- 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
- 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
- 2018Do Better ImageNet Models Transfer Better?cited by 560
- 2019The State of Sparsity in Deep Neural Networkscited by 471
- 2018A Survey on Deep Transfer Learningcited by 417
- 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
- 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
- 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
- 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
- 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 182
- 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 182
- 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
- 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
- 2019Transferability and Hardness of Supervised Classification Taskscited by 125
- 2020What makes instance discrimination good for transfer learning?cited by 111
- 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
- 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98
- 2021A Broad Study on the Transferability of Visual Representations with Contrastive Learningcited by 78
- 2022Visual Prompt Tuning for Generative Transfer Learningcited by 75
- 2021OTCE: A Transferability Metric for Cross-Domain Cross-Task Representationscited by 62
- 2021Transferability Estimation using Bhattacharyya Class Separabilitycited by 53
- 2019Densely Connected Search Space for More Flexible Neural Architecture Searchcited by 52
- 2019Deep Model Transferability from Attribution Mapscited by 28
- 2020Which Model to Transfer? Finding the Needle in the Growing Haystackcited by 21
- 2020DEPARA: Deep Attribution Graph for Deep Knowledge Transferabilitycited by 18
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 1; 2025: 1.
3 papers.