# How transfer learning impacts linguistic knowledge in deep NLP models?

arXiv: https://arxiv.org/abs/2105.15179 · HTML: https://mlsanity.com/paper/2105.15179/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 31 May 2021.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [What Happens To BERT Embeddings During Fine-tuning?](https://mlsanity.com/paper/2004.14448/) (2020)
- [Identifying and Controlling Important Neurons in Neural Machine Translation](https://mlsanity.com/paper/1811.01157/) (2018)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks](https://mlsanity.com/paper/1801.07772/) (2018)
- [Similarity Analysis of Contextual Word Representation Models](https://mlsanity.com/paper/2005.01172/) (2020)
- [The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations](https://mlsanity.com/paper/1702.03964/) (2017)
- [On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers](https://mlsanity.com/paper/2010.02616/) (2020)
- [On the Linguistic Representational Power of Neural Machine Translation Models](https://mlsanity.com/paper/1911.00317/) (2019)
- [NeuroX: A Toolkit for Analyzing Individual Neurons in Neural Networks](https://mlsanity.com/paper/1812.09359/) (2018)
- [Word Representation Models for Morphologically Rich Languages in Neural Machine Translation](https://mlsanity.com/paper/1606.04217/) (2016)

## Built on by (most cited first, 30 of 32)

- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Discovering Latent Concepts Learned in BERT](https://mlsanity.com/paper/2205.07237/) (2022)
- [Analyzing Encoded Concepts in Transformer Language Models](https://mlsanity.com/paper/2206.13289/) (2022)
- [Emergent Modularity in Pre-trained Transformers](https://mlsanity.com/paper/2305.18390/) (2023)
- [How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy](https://mlsanity.com/paper/2109.04740/) (2021)
- [Speech Translation with Speech Foundation Models and Large Language Models: What is There and What is Missing?](https://mlsanity.com/paper/2402.12025/) (2024)
- [On the Transformation of Latent Space in Fine-Tuned NLP Models](https://mlsanity.com/paper/2210.12696/) (2022)
- [Can LLMs facilitate interpretation of pre-trained language models?](https://mlsanity.com/paper/2305.13386/) (2023)
- [Predicting Fine-Tuning Performance with Probing](https://mlsanity.com/paper/2210.07352/) (2022)
- [Exploring Alignment in Shared Cross-lingual Spaces](https://mlsanity.com/paper/2405.14535/) (2024)
- [On the Importance of Data Size in Probing Fine-tuned Models](https://mlsanity.com/paper/2203.09627/) (2022)
- [HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation](https://mlsanity.com/paper/2212.08853/) (2022)
- [NeuroX Library for Neuron Analysis of Deep NLP Models](https://mlsanity.com/paper/2305.17073/) (2023)
- [ArmanEmo: A Persian Dataset for Text-based Emotion Detection](https://mlsanity.com/paper/2207.11808/) (2022)
- [Scaling up Discovery of Latent Concepts in Deep NLP Models](https://mlsanity.com/paper/2308.10263/) (2023)
- [Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain](https://mlsanity.com/paper/2311.07766/) (2023)
- [Transformers and the representation of biomedical background knowledge](https://mlsanity.com/paper/2202.02432/) (2022)
- [Can BERT eat RuCoLA? Topological Data Analysis to Explain](https://mlsanity.com/paper/2304.01680/) (2023)
- [Interpretability in Activation Space Analysis of Transformers: A Focused Survey](https://mlsanity.com/paper/2302.09304/) (2023)
- [CoNTACT: A Dutch COVID-19 Adapted BERT for Vaccine Hesitancy and Argumentation Detection](https://mlsanity.com/paper/2203.07362/) (2022)
- [Persian Emotion Detection using ParsBERT and Imbalanced Data Handling Approaches](https://mlsanity.com/paper/2211.08029/) (2022)
- [Beyond the Leaderboard: Understanding Performance Disparities in Large Language Models via Model Diffing](https://mlsanity.com/paper/2509.18792/) (2025)
- [Improving Similar Language Translation With Transfer Learning](https://mlsanity.com/paper/2108.03533/) (2021)
- [Post-hoc analysis of Arabic transformer models](https://mlsanity.com/paper/2210.09990/) (2022)
- [Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?](https://mlsanity.com/paper/2301.11688/) (2023)
- [Statistical Deficiency for Task Inclusion Estimation](https://mlsanity.com/paper/2503.05491/) (2025)
- [Improve Language Model and Brain Alignment via Associative Memory](https://mlsanity.com/paper/2505.13844/) (2025)
- [A Representation Level Analysis of NMT Model Robustness to Grammatical Errors](https://mlsanity.com/paper/2505.21224/) (2025)
- [From Words to Waves: Analyzing Concept Formation in Speech and Text-Based Foundation Models](https://mlsanity.com/paper/2506.01133/) (2025)
- [The (In)Effectiveness of Intermediate Task Training For Domain Adaptation and Cross-Lingual Transfer Learning](https://mlsanity.com/paper/2210.01091/) (2022)

## Abstract (arXiv)

Transfer learning from pre-trained neural language models towards downstream tasks has been a predominant theme in NLP recently. Several researchers have shown that deep NLP models learn non-trivial amount of linguistic knowledge, captured at different layers of the model. We investigate how fine-tuning towards downstream NLP tasks impacts the learned linguistic knowledge. We carry out a study across popular pre-trained models BERT, RoBERTa and XLNet using layer and neuron-level diagnostic classifiers. We found that for some GLUE tasks, the network relies on the core linguistic information and preserve it deeper in the network, while for others it forgets. Linguistic information is distributed in the pre-trained language models but becomes localized to the lower layers post fine-tuning, reserving higher layers for the task specific knowledge. The pattern varies across architectures, with BERT retaining linguistic information relatively deeper in the network compared to RoBERTa and XLNet, where it is predominantly delegated to the lower layers.