# Investigating Transferability in Pretrained Language Models

arXiv: https://arxiv.org/abs/2004.14975 · HTML: https://mlsanity.com/paper/2004.14975/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2020.

## Built on by (most cited first, 28 of 28)

- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [Contrastive Explanations for Model Interpretability](https://mlsanity.com/paper/2103.01378/) (2021)
- [Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction](https://mlsanity.com/paper/2105.06965/) (2021)
- [Scaling Laws for Downstream Task Performance of Large Language Models](https://mlsanity.com/paper/2402.04177/) (2024)
- [Sparse Interventions in Language Models with Differentiable Masking](https://mlsanity.com/paper/2112.06837/) (2021)
- [Grounding Representation Similarity with Statistical Testing](https://mlsanity.com/paper/2108.01661/) (2021)
- [NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Tasks](https://mlsanity.com/paper/2306.03208/) (2023)
- [On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies](https://mlsanity.com/paper/2104.05694/) (2021)
- [Rethinking Why Intermediate-Task Fine-Tuning Works](https://mlsanity.com/paper/2108.11696/) (2021)
- [Diagnosing AI Explanation Methods with Folk Concepts of Behavior](https://mlsanity.com/paper/2201.11239/) (2022)
- [Hidden State Variability of Pretrained Language Models Can Guide Computation Reduction for Transfer Learning](https://mlsanity.com/paper/2210.10041/) (2022)
- [What's in your Head? Emergent Behaviour in Multi-Task Transformer Models](https://mlsanity.com/paper/2104.06129/) (2021)
- [Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies](https://mlsanity.com/paper/2202.12312/) (2022)
- [Damage detection using in-domain and cross-domain transfer learning](https://mlsanity.com/paper/2102.03858/) (2021)
- [The Rediscovery Hypothesis: Language Models Need to Meet Linguistics](https://mlsanity.com/paper/2103.01819/) (2021)
- [REPT: Bridging Language Models and Machine Reading Comprehension via Retrieval-Based Pre-training](https://mlsanity.com/paper/2105.04201/) (2021)
- [Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks](https://mlsanity.com/paper/2111.01340/) (2021)
- [ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learning](https://mlsanity.com/paper/2205.05282/) (2022)
- [PEACE: Cross-Platform Hate Speech Detection- A Causality-guided Framework](https://mlsanity.com/paper/2306.08804/) (2023)
- [Probing the Category of Verbal Aspect in Transformer Language Models](https://mlsanity.com/paper/2406.02335/) (2024)
- [Antibody Representation Learning for Drug Discovery](https://mlsanity.com/paper/2210.02881/) (2022)
- [Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models](https://mlsanity.com/paper/2104.08410/) (2021)
- [What do Transformers Know about Government?](https://mlsanity.com/paper/2404.14270/) (2024)
- [Portable Semantics, Private Dialects: Reuse and Negative Transfer in Latent Communication Between Language-Model Cells](https://mlsanity.com/paper/2609.11365/) (2026)
- [The Effect of Masking Strategies on Knowledge Retention by Language Models](https://mlsanity.com/paper/2306.07185/) (2023)
- [Disentangling the Linguistic Competence of Privacy-Preserving BERT](https://mlsanity.com/paper/2310.11363/) (2023)

## Abstract (arXiv)

How does language model pretraining help transfer learning? We consider a simple ablation technique for determining the impact of each pretrained layer on transfer task performance. This method, partial reinitialization, involves replacing different layers of a pretrained model with random weights, then finetuning the entire model on the transfer task and observing the change in performance. This technique reveals that in BERT, layers with high probing performance on downstream GLUE tasks are neither necessary nor sufficient for high accuracy on those tasks. Furthermore, the benefit of using pretrained parameters for a layer varies dramatically with finetuning dataset size: parameters that provide tremendous performance improvement when data is plentiful may provide negligible benefits in data-scarce settings. These results reveal the complexity of the transfer learning process, highlighting the limitations of methods that operate on frozen models or single data samples.