# What Happens To BERT Embeddings During Fine-tuning?

arXiv: https://arxiv.org/abs/2004.14448 · HTML: https://mlsanity.com/paper/2004.14448/

## Key facts

- Cited by 113 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 30 of 113)

- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://mlsanity.com/paper/2402.14811/) (2024)
- [On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation](https://mlsanity.com/paper/2106.03164/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT](https://mlsanity.com/paper/2101.11109/) (2021)
- [Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks](https://mlsanity.com/paper/2112.03271/) (2021)
- [A Closer Look at How Fine-tuning Changes BERT](https://mlsanity.com/paper/2106.14282/) (2021)
- [Who's asking? User personas and the mechanics of latent misalignment](https://mlsanity.com/paper/2406.12094/) (2024)
- [Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuning](https://mlsanity.com/paper/2504.02922/) (2025)
- [How transfer learning impacts linguistic knowledge in deep NLP models?](https://mlsanity.com/paper/2105.15179/) (2021)
- [Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers](https://mlsanity.com/paper/2109.08406/) (2021)
- [CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks](https://mlsanity.com/paper/2112.02714/) (2021)
- [Discovering Language-neutral Sub-networks in Multilingual Language Models](https://mlsanity.com/paper/2205.12672/) (2022)
- [Narrow Finetuning Leaves Clearly Readable Traces in Activation Differences](https://mlsanity.com/paper/2510.13900/) (2025)
- [DirectProbe: Studying Representations without Classifiers](https://mlsanity.com/paper/2104.05904/) (2021)
- [Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models](https://mlsanity.com/paper/2305.16597/) (2023)
- [How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy](https://mlsanity.com/paper/2109.04740/) (2021)
- [Investigating Forgetting in Pre-Trained Representations Through Continual Learning](https://mlsanity.com/paper/2305.05968/) (2023)
- [Towards Efficient Fine-tuning of Pre-trained Code Models: An Experimental Study and Beyond](https://mlsanity.com/paper/2304.05216/) (2023)
- [Acceptability Judgements via Examining the Topology of Attention Maps](https://mlsanity.com/paper/2205.09630/) (2022)
- [On the Transformation of Latent Space in Fine-Tuned NLP Models](https://mlsanity.com/paper/2210.12696/) (2022)
- [ReSi: A Comprehensive Benchmark for Representational Similarity Measures](https://mlsanity.com/paper/2408.00531/) (2024)
- [Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models](https://mlsanity.com/paper/2402.19465/) (2024)
- [Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representations](https://mlsanity.com/paper/2109.05958/) (2021)
- [On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learning](https://mlsanity.com/paper/2212.09170/) (2022)

## Abstract (arXiv)

While there has been much recent work studying how linguistic information is encoded in pre-trained sentence representations, comparatively little is understood about how these models change when adapted to solve downstream tasks. Using a suite of analysis techniques (probing classifiers, Representational Similarity Analysis, and model ablations), we investigate how fine-tuning affects the representations of the BERT model. We find that while fine-tuning necessarily makes significant changes, it does not lead to catastrophic forgetting of linguistic phenomena. We instead find that fine-tuning primarily affects the top layers of BERT, but with noteworthy variation across tasks. In particular, dependency parsing reconfigures most of the model, whereas SQuAD and MNLI appear to involve much shallower processing. Finally, we also find that fine-tuning has a weaker effect on representations of out-of-domain sentences, suggesting room for improvement in model generalization.