mlsanityData to 14 Oct 2026

Where to start? Analyzing the potential value of intermediate models

arXiv 2211.00107 ↗First posted 31 Oct 2022cs.CL cs.AI cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Previous studies observed that finetuned models may be better base models than the vanilla pretrained model. Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset. Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks. Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point. This is in contrast to current perception that the alignment between the target dataset and the source dataset used to generate the base model is a major factor in determining intertraining success. We analyze different aspects that contribute to each. Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings. Last, we release an updating ranking of best models in the HuggingFace hub per architecture https://ibm.github.io/model-recycling/.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 48 of its 77 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

48 of 77 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  11. 2018Datasheets for Datasetscited by 1,386
  12. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
36 more
  1. 2018Model Cards for Model Reportingcited by 1,080
  2. 2018Neural Network Acceptability Judgmentscited by 958
  3. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  4. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  5. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  6. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  7. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 417
  8. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  9. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 271
  10. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 215
  11. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
  12. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  13. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 170
  14. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  15. 2020Exploring and Predicting Transferability across NLP Taskscited by 128
  16. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  17. 2022Fusing finetuned models for better pretrainingcited by 110
  18. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  19. 2018The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standardscited by 97
  20. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  21. 2020Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approachcited by 83
  22. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  23. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 63
  24. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  25. 2022Linear Connectivity Reveals Generalization Strategiescited by 56
  26. 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
  27. 2017Emotion Intensities in Tweetscited by 32
  28. 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
  29. 2021Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cardscited by 24
  30. 2021The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLPcited by 19
  31. 2021Rethinking Why Intermediate-Task Fine-Tuning Workscited by 18
  32. 2022When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learningcited by 17
  33. 2022Cluster & Tune: Boost Cold Start Performance in Text Classificationcited by 13
  34. 2022PreQuEL: Quality Estimation of Machine Translation Outputs in Advancecited by 9
  35. 2022Evidence > Intuition: Transferability Estimation for Encoder Selectioncited by 7
  36. 2022Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysiscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 11; 2024: 7; 2025: 1; 2026: 2.

23 papers.

Sort:

By area:

  1. 2024Model merging with SVD to tie the Knotscited by 122
  2. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  3. 2023Deep Model Fusion: A Surveycited by 91
  4. 2023Soft Merging of Experts with Adaptive Routingcited by 90
  5. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  6. 2023Efficient Benchmarking of Language Modelscited by 71
  7. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  8. 2024What Matters for Model Merging at Scale?cited by 55
  9. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  10. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  11. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  12. 2023Git-Theta: A Git Extension for Collaborative Development of Machine Learning Modelscited by 15
11 more
  1. 2024Evaluation of Geographical Distortions in Language Modelscited by 6
  2. 2023QAID: Question Answering Inspired Few-shot Intent Detectioncited by 5
  3. 2024Local Superior Soups: A Catalyst for Model Merging in Cross-Silo Federated Learningcited by 5
  4. 2023Competence-Based Analysis of Language Modelscited by 4
  5. 2023How Different Is Stereotypical Bias Across Languages?cited by 4
  6. 2024IDoFew: Intermediate Training Using Dual-Clustering in Language Models for Few Labels Text Classificationcited by 4
  7. 2026Will it Merge? On The Causes of Model Mergeabilitycited by 4
  8. 2023Towards Robust and Efficient Continual Language Learningcited by 2
  9. 2023Leveraging Few-Shot Data Augmentation and Waterfall Prompting for Response Generationcited by 1
  10. 2023RSVP: Customer Intent Detection via Agent Response Contrastive and Generative Pre-Trainingcited by 1
  11. 2026Discovering Hidden Gems in Model Repositoriescited by 1