mlsanityData to 7 Oct 2026

Rethinking Why Intermediate-Task Fine-Tuning Works

arXiv 2108.11696 ↗First posted 26 Aug 2021cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Supplementary Training on Intermediate Labeled-data Tasks (STILTs) is a widely applied technique, which first fine-tunes the pretrained language models on an intermediate task before on the target task of interest. While STILTs is able to further improve the performance of pretrained language models, it is still unclear why and when it works. Previous research shows that those intermediate tasks involving complex inference, such as commonsense reasoning, work especially well for RoBERTa. In this paper, we discover that the improvement from an intermediate task could be orthogonal to it containing reasoning or other complex skills -- a simple real-fake discrimination task synthesized by GPT2 can benefit diverse target tasks. We conduct extensive experiments to study the impact of different factors on STILTs. These findings suggest rethinking the role of intermediate fine-tuning in the STILTs pipeline.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 26 Aug 2021.

Builds on

25 of 38 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  8. 2018Neural Network Acceptability Judgmentscited by 952
  9. 2017Dense-Captioning Events in Videoscited by 789
  10. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  12. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
13 more
  1. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  2. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  3. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  4. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  5. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  6. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  7. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  8. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  9. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  10. 2020Rethinking embedding coupling in pre-trained language modelscited by 103
  11. 2020English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Toocited by 39
  12. 2020Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoningcited by 33
  13. 2020Investigating Transferability in Pretrained Language Modelscited by 28

Built on by

Cited by year

Citing papers per year: 2022: 8; 2023: 5; 2024: 2; 2025: 3.

18 papers.

Sort:

  1. 2022Fusing finetuned models for better pretrainingcited by 110
  2. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  3. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  4. 2022Where to start? Analyzing the potential value of intermediate modelscited by 23
  5. 2022When to Use Multi-Task Learning vs Intermediate Fine-Tuning for Pre-Trained Encoder Transfer Learningcited by 17
  6. 2023Cross-lingual Cross-temporal Summarization: Dataset, Models, Evaluationcited by 8
  7. 2024MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarizationcited by 8
  8. 2022Scope of Pre-trained Language Models for Detecting Conflicting Health Informationcited by 4
  9. 2023Towards Robust and Efficient Continual Language Learningcited by 2
  10. 2025Latent Traits and Cross-Task Transfer: Deconstructing Dataset Interactions in LLM Fine-tuningcited by 2
  11. 2023ViM: Vision Middleware for Unified Downstream Transferringcited by 1
  12. 2023IEKG: A Commonsense Knowledge Graph for Idiomatic Expressionscited by 1
6 more
  1. 2022Schema-Free Dependency Parsing via Sequence Generationcited by 0
  2. 2022Incorporating Emotions into Health Mention Classification Task on Social Mediacited by 0
  3. 2023A Framework for Identifying Depression on Social Media: MentalRiskES@IberLEF 2023cited by 0
  4. 2024Linguistic Knowledge Can Enhance Encoder-Decoder Models (If You Let It)cited by 0
  5. 2025Enhancing Health Mention Classification Performance: A Study on Advancements in Parameter Efficient Tuningcited by 0
  6. 2025Diminishing Returns in Self-Supervised Learningcited by 0