# Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations

arXiv: https://arxiv.org/abs/2104.10833 · HTML: https://mlsanity.com/paper/2104.10833/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 33 references that are in the map.
- First posted on arXiv 22 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://mlsanity.com/paper/2002.01808/) (2020)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [Simple and Effective Dimensionality Reduction for Word Embeddings](https://mlsanity.com/paper/1708.03629/) (2017)
- [Evaluating Word Embedding Models: Methods and Experimental Results](https://mlsanity.com/paper/1901.09785/) (2019)
- [Cracking the Contextual Commonsense Code: Understanding Commonsense Reasoning Aptitude of Deep Contextual Representations](https://mlsanity.com/paper/1910.01157/) (2019)
- [Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity](https://mlsanity.com/paper/1909.02339/) (2019)
- [Specialising Word Vectors for Lexical Entailment](https://mlsanity.com/paper/1710.06371/) (2017)
- [Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge](https://mlsanity.com/paper/2010.13057/) (2020)
- [Extrofitting: Enriching Word Representation and its Vector Space with Semantic Lexicons](https://mlsanity.com/paper/1804.07946/) (2018)

## Built on by (most cited first, 13 of 13)

- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Stable Anisotropic Regularization](https://mlsanity.com/paper/2305.19358/) (2023)
- [Reliable Measures of Spread in High Dimensional Latent Spaces](https://mlsanity.com/paper/2212.08172/) (2022)
- [Is Anisotropy Inherent to Transformers?](https://mlsanity.com/paper/2306.07656/) (2023)
- [Learning Shortcuts: On the Misleading Promise of NLU in Language Models](https://mlsanity.com/paper/2401.09615/) (2024)
- [Uncovering the Latent Potential of Deep Intermediate Representations](https://mlsanity.com/paper/2605.23033/) (2026)
- [Calibration Error Estimation Using Fuzzy Binning](https://mlsanity.com/paper/2305.00543/) (2023)
- [Exploring the Impact of a Transformer's Latent Space Geometry on Downstream Task Performance](https://mlsanity.com/paper/2406.12159/) (2024)
- [Interpretable Privacy Preservation of Text Representations Using Vector Steganography](https://mlsanity.com/paper/2112.02557/) (2021)
- [On Information Hiding in Natural Language Systems](https://mlsanity.com/paper/2203.06512/) (2022)
- [Evaluating Language Models on Grooming Risk Estimation Using Fuzzy Theory](https://mlsanity.com/paper/2502.12563/) (2025)
- [A Fuzzy Evaluation of Sentence Encoders on Grooming Risk Classification](https://mlsanity.com/paper/2502.12576/) (2025)
- [Isotropy Cliffs: The Geometric Signature of Decision-Making in Large Language Models](https://mlsanity.com/paper/2608.00828/) (2026)

## Abstract (arXiv)

Contextual word representation models have shown massive improvements on a multitude of NLP tasks, yet their word sense disambiguation capabilities remain poorly explained. To address this gap, we assess whether contextual word representations extracted from deep pretrained language models create distinguishable representations for different senses of a given word. We analyze the representation geometry and find that most layers of deep pretrained language models create highly anisotropic representations, pointing towards the existence of representation degeneration problem in contextual word representations. After accounting for anisotropy, our study further reveals that there is variability in sense learning capabilities across different language models. Finally, we propose LASeR, a 'Low Anisotropy Sense Retrofitting' approach that renders off-the-shelf representations isotropic and semantically more meaningful, resolving the representation degeneration problem as a post-processing step, and conducting sense-enrichment of contextualized representations extracted from deep neural language models.