Exploiting a Zoo of Checkpoints for Unseen Tasks
Not on the map. Why?
Abstract (arXiv)
There are so many models in the literature that it is difficult for practitioners to decide which combinations are likely to be effective for a new task. This paper attempts to address this question by capturing relationships among checkpoints published on the web. We model the space of tasks as a Gaussian process. The covariance can be estimated from checkpoints and unlabeled probing data. With the Gaussian process, we can identify representative checkpoints by a maximum mutual information criterion. This objective is submodular. A greedy method identifies representatives that are likely to "cover" the task space. These representatives generalize to new tasks with superior performance. Empirical evidence is provided for applications from both computational linguistics as well as computer vision.
Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 5 Nov 2021.
Builds on
25 of 38 references.
References · 25
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
- 2020Language Models are Few-Shot Learnerscited by 32,320
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,870
- 2020Longformer: The Long-Document Transformercited by 3,393
- 2017Prototypical Networks for Few-shot Learningcited by 3,011
- 2016Matching Networks for One Shot Learningcited by 2,251
- 2019Similarity of Neural Network Representations Revisitedcited by 1,719
- 2018Taskonomy: Disentangling Task Transfer Learningcited by 633
- 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
- 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 242
- 2018Characterizing and Avoiding Negative Transfercited by 227
- 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
13 more
- 2019Task2Vec: Task Embedding for Meta-Learningcited by 201
- 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
- 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 183
- 2020Geometric Dataset Distances via Optimal Transportcited by 158
- 2020Exploring and Predicting Transferability across NLP Taskscited by 126
- 2019Transferability and Hardness of Supervised Classification Taskscited by 125
- 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
- 2017Meta-Learning by Adjusting Priors Based on Extended PAC-Bayes Theorycited by 76
- 2018Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?cited by 62
- 2020Ranking Neural Checkpointscited by 50
- 2019Deep Model Transferability from Attribution Mapscited by 28
- 2018Neural models of factualitycited by 20
- 2016Semantic Tagging with Deep Residual Networkscited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.