# Learning to Remove: Towards Isotropic Pre-trained BERT Embedding

arXiv: https://arxiv.org/abs/2104.05274 · HTML: https://mlsanity.com/paper/2104.05274/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 12 Apr 2021.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [FRAGE: Frequency-Agnostic Word Representation](https://mlsanity.com/paper/1809.06858/) (2018)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Conceptor Debiasing of Word Representations Evaluated on WEAT](https://mlsanity.com/paper/1906.05993/) (2019)
- [IsoBN: Fine-Tuning BERT with Isotropic Batch Normalization](https://mlsanity.com/paper/2005.02178/) (2020)
- [Unsupervised Post-processing of Word Vectors via Conceptor Negation](https://mlsanity.com/paper/1811.11001/) (2018)

## Built on by (most cited first, 23 of 23)

- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [Outliers Dimensions that Disrupt Transformers Are Driven by Frequency](https://mlsanity.com/paper/2205.11380/) (2022)
- [IsoScore: Measuring the Uniformity of Embedding Space Utilization](https://mlsanity.com/paper/2108.07344/) (2021)
- [Improving Position Encoding of Transformers for Multivariate Time Series Classification](https://mlsanity.com/paper/2305.16642/) (2023)
- [Using Sequences of Life-events to Predict Human Lives](https://mlsanity.com/paper/2306.03009/) (2023)
- [Byte Pair Encoding for Symbolic Music](https://mlsanity.com/paper/2301.11975/) (2023)
- [Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity](https://mlsanity.com/paper/2306.00458/) (2023)
- [Outlier Dimensions Encode Task-Specific Knowledge](https://mlsanity.com/paper/2310.17715/) (2023)
- [Stable Anisotropic Regularization](https://mlsanity.com/paper/2305.19358/) (2023)
- [Transformer Language Models Handle Word Frequency in Prediction Head](https://mlsanity.com/paper/2305.18294/) (2023)
- [Impact of time and note duration tokenizations on deep learning symbolic music modeling](https://mlsanity.com/paper/2310.08497/) (2023)
- [Reliable Measures of Spread in High Dimensional Latent Spaces](https://mlsanity.com/paper/2212.08172/) (2022)
- [Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings](https://mlsanity.com/paper/2406.10984/) (2024)
- [Isotropy, Clusters, and Classifiers](https://mlsanity.com/paper/2402.03191/) (2024)
- [Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics](https://mlsanity.com/paper/2604.08764/) (2026)
- [Generative Giants, Retrieval Weaklings: Why do Multimodal Large Language Models Fail at Multimodal Retrieval?](https://mlsanity.com/paper/2512.19115/) (2025)
- [SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models](https://mlsanity.com/paper/2606.32022/) (2026)
- [Rethinking ASTE: A Minimalist Tagging Scheme Alongside Contrastive Learning](https://mlsanity.com/paper/2403.07342/) (2024)
- [Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance](https://mlsanity.com/paper/2406.12159/) (2024)
- [Norm of Mean Contextualized Embeddings Determines their Variance](https://mlsanity.com/paper/2409.11253/) (2024)
- [ALF: Advertiser Large Foundation Model for Multi-Modal Advertiser Understanding](https://mlsanity.com/paper/2504.18785/) (2025)
- [Vec2Summ: Text Summarization via Probabilistic Sentence Embeddings](https://mlsanity.com/paper/2508.07017/) (2025)
- [Where A Small Language Model Helps in Invoice Categorisation, Understood Through Embedding Geometry](https://mlsanity.com/paper/2608.18033/) (2026)

## Abstract (arXiv)

Pre-trained language models such as BERT have become a more common choice of natural language processing (NLP) tasks. Research in word representation shows that isotropic embeddings can significantly improve performance on downstream tasks. However, we measure and analyze the geometry of pre-trained BERT embedding and find that it is far from isotropic. We find that the word vectors are not centered around the origin, and the average cosine similarity between two random words is much higher than zero, which indicates that the word vectors are distributed in a narrow cone and deteriorate the representation capacity of word embedding. We propose a simple, and yet effective method to fix this problem: remove several dominant directions of BERT embedding with a set of learnable weights. We train the weights on word similarity tasks and show that processed embedding is more isotropic. Our method is evaluated on three standardized tasks: word similarity, word analogy, and semantic textual similarity. In all tasks, the word embedding processed by our method consistently outperforms the original embedding (with average improvement of 13% on word analogy and 16% on semantic textual similarity) and two baseline methods. Our method is also proven to be more robust to changes of hyperparameter.