mlsanityData to 7 Oct 2026

How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey

arXiv 2312.04775 ↗First posted 8 Dec 2023cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Transferability estimation has been attached to great attention in the computer vision fields. Researchers try to estimate with low computational cost the performance of a model when transferred from a source task to a given target task. Considering the effectiveness of such estimations, the communities of natural language processing also began to study similar problems for the selection of pre-trained language models. However, there is a lack of a comprehensive comparison between these estimation methods yet. Also, the differences between vision and language scenarios make it doubtful whether previous conclusions can be established across fields. In this paper, we first conduct a thorough survey of existing transferability estimation methods being able to find the most suitable model, then we conduct a detailed empirical study for the surveyed methods based on the GLUE benchmark. From qualitative and quantitative analyses, we demonstrate the strengths and weaknesses of existing methods and show that H-Score generally performs well with superiorities in effectiveness and efficiency. We also outline the difficulties of consideration of training details, applicability to text generation, and consistency to certain metrics which shed light on future directions.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 33 of its 46 references that are in the map. First posted on arXiv 8 Dec 2023.

Builds on

33 of 46 references.

References · 33

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  7. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  10. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  11. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  12. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
21 more
  1. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  2. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 168
  3. 2019PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variablecited by 145
  4. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  5. 2019Transferability and Hardness of Supervised Classification Taskscited by 125
  6. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  7. 2021Ensembling Off-the-shelf Models for GAN Trainingcited by 84
  8. 2022An Information-Theoretic Approach to Transferability in Task Transfer Learningcited by 79
  9. 2022A Review of Deep Transfer Learning and Recent Advancementscited by 56
  10. 2020Ranking Neural Checkpointscited by 50
  11. 2020Scalable Transfer Learning with Expert Modelscited by 48
  12. 2021Scalable Diverse Model Selection for Accessible Transfer Learningcited by 40
  13. 2020A Survey on Transfer Learning in Natural Language Processingcited by 33
  14. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  15. 2022How stable are Transferability Metrics evaluations?cited by 30
  16. 2022Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Spacecited by 30
  17. 2020Duality Diagram Similarity: a generic framework for initialization selection in task transfer learningcited by 23
  18. 2022Threats to Pre-trained Language Models: Survey and Taxonomycited by 16
  19. 2021Graph-Based Similarity of Neural Network Representationscited by 7
  20. 2022Evidence > Intuition: Transferability Estimation for Encoder Selectioncited by 7
  21. 2021Exploiting a Zoo of Checkpoints for Unseen Taskscited by 3

Built on by

Cited by year

Citing papers per year: 2024: 4.

4 papers.

  1. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  2. 2024Which Model to Transfer? A Survey on Transferability Estimationcited by 13
  3. 2024A Strategy to Combine 1stGen Transformers and Open LLMs for Automatic Text Classificationcited by 3
  4. 2024Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Rankingcited by 0