mlsanityData to 7 Oct 2026

How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy

arXiv 2109.04740 ↗First posted 10 Sep 2021cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

It is widely accepted that fine-tuning pre-trained language models usually brings about performance improvements in downstream tasks. However, there are limited studies on the reasons behind this effectiveness, particularly from the viewpoint of structural changes in the embedding space. Trying to fill this gap, in this paper, we analyze the extent to which the isotropy of the embedding space changes after fine-tuning. We demonstrate that, even though isotropy is a desirable geometrical property, fine-tuning does not necessarily result in isotropy enhancements. Moreover, local structures in pre-trained contextual word representations (CWRs), such as those encoding token types or frequency, undergo a massive change during fine-tuning. Our experiments show dramatic growth in the number of elongated directions in the embedding space, which, in contrast to pre-trained CWRs, carry the essential linguistic knowledge in the fine-tuned embedding space, making existing isotropy enhancement methods ineffective.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

29 of 38 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  7. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  8. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  9. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  10. 2016Using the Output Embedding to Improve Language Modelscited by 266
  11. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  12. 2019Representation Degeneration Problem in Training Natural Language Generation Modelscited by 250
17 more
  1. 2019Visualizing and Measuring the Geometry of BERTcited by 215
  2. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 205
  3. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  4. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  5. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  6. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  7. 2016Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modelingcited by 87
  8. 2021Probing BERT in Hyperbolic Spacescited by 50
  9. 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
  10. 2020On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformerscited by 39
  11. 2020Probing Linguistic Systematicitycited by 37
  12. 2020Asking without Telling: Exploring Latent Ontologies in Contextual Representationscited by 34
  13. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  14. 2021A Cluster-based Approach for Improving Isotropy in Contextual Embedding Spacecited by 27
  15. 2021DirectProbe: Studying Representations without Classifierscited by 25
  16. 2020Infusing Finetuning with Semantic Dependenciescited by 18
  17. 2021On the Interplay Between Fine-tuning and Composition in Transformerscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 3; 2024: 7; 2025: 4; 2026: 4.

23 papers.

Sort:

By area:

  1. 2022Outliers Dimensions that Disrupt Transformers Are Driven by Frequencycited by 55
  2. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  3. 2022Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropizationcited by 19
  4. 2023Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similaritycited by 17
  5. 2022On Isotropy, Contextualization and Learning Dynamics of Contrastive-based Sentence Representation Learningcited by 15
  6. 2023Stable Anisotropic Regularizationcited by 12
  7. 2024Voices in a Crowd: Searching for Clusters of Unique Perspectivescited by 9
  8. 2024Extracting Sentence Embeddings from Pretrained Transformer Modelscited by 7
  9. 2022Transformers and the representation of biomedical background knowledgecited by 5
  10. 2022Reliable Measures of Spread in High Dimensional Latent Spacescited by 5
  11. 2023What's left can't be right -- The remaining positional incompetence of contrastive vision-language modelscited by 5
  12. 2024Isotropy, Clusters, and Classifierscited by 4
11 more
  1. 2024Isotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Searchcited by 2
  2. 2024Investigating Distributions of Telecom Adapted Sentence Embeddings for Document Retrievalcited by 1
  3. 2024Whitening Not Recommended for Classification Tasks in LLMscited by 1
  4. 2025Assessing the Impact of Anisotropy in Neural Representations of Speech: A Case Study on Keyword Spottingcited by 1
  5. 2025When Annotators Disagree, Topology Explains: Mapper, a Topological Tool for Exploring Text Embedding Geometry and Ambiguitycited by 1
  6. 2025Freeze, Diffuse, Decode: Task-Aware Adaptation of Transformer Embeddings for Antimicrobial Peptide Designcited by 0
  7. 2025A framework for auditing grounding claimscited by 0
  8. 2026Latent Knowledge as a Predictor of Fact Acquisition in Fine-Tuned Large Language Modelscited by 0
  9. 2026Towards Faster Language Model Inference Using Mixture-of-Experts Flow Matchingcited by 0
  10. 2026Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometrycited by 0
  11. 2026The Geometry of Low-Resource Language Representationscited by 0