# Shades of meaning: Uncovering the geometry of ambiguous word representations through contextualised language models

arXiv: https://arxiv.org/abs/2304.13597 · HTML: https://mlsanity.com/paper/2304.13597/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 56 references that are in the map.
- First posted on arXiv 26 Apr 2023.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Regularizing and Optimizing LSTM Language Models](https://mlsanity.com/paper/1708.02182/) (2017)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [From Word to Sense Embeddings: A Survey on Vector Representations of Meaning](https://mlsanity.com/paper/1805.04032/) (2018)
- [Why Do Masked Neural Language Models Still Need Common Sense Knowledge?](https://mlsanity.com/paper/1911.03024/) (2019)
- [Extending Machine Language Models toward Human-Level Language Understanding](https://mlsanity.com/paper/1912.05877/) (2019)
- [Geometry of Polysemy](https://mlsanity.com/paper/1610.07569/) (2016)

## Built on by (most cited first, 2 of 2)

- [Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding](https://mlsanity.com/paper/2310.19671/) (2023)
- [BERT's Conceptual Cartography: Mapping the Landscapes of Meaning](https://mlsanity.com/paper/2408.07190/) (2024)

## Abstract (arXiv)

Lexical ambiguity presents a profound and enduring challenge to the language sciences. Researchers for decades have grappled with the problem of how language users learn, represent and process words with more than one meaning. Our work offers new insight into psychological understanding of lexical ambiguity through a series of simulations that capitalise on recent advances in contextual language models. These models have no grounded understanding of the meanings of words at all; they simply learn to predict words based on the surrounding context provided by other words. Yet, our analyses show that their representations capture fine-grained meaningful distinctions between unambiguous, homonymous, and polysemous words that align with lexicographic classifications and psychological theorising. These findings provide quantitative support for modern psychological conceptualisations of lexical ambiguity and raise new challenges for understanding of the way that contextual information shapes the meanings of words across different timescales.