# Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge

arXiv: https://arxiv.org/abs/2010.13057 · HTML: https://mlsanity.com/paper/2010.13057/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Oct 2020.

## Built on by (most cited first, 18 of 18)

- [minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Models](https://mlsanity.com/paper/2203.13112/) (2022)
- [Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations](https://mlsanity.com/paper/2104.10833/) (2021)
- [Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics](https://mlsanity.com/paper/2311.01386/) (2023)
- [RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English)](https://mlsanity.com/paper/2105.13266/) (2021)
- [Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy](https://mlsanity.com/paper/2106.13553/) (2021)
- [HumBEL: A Human-in-the-Loop Approach for Evaluating Demographic Factors of Language Models in Human-Machine Conversations](https://mlsanity.com/paper/2305.14195/) (2023)
- [Transformers, Contextualism, and Polysemy](https://mlsanity.com/paper/2404.09577/) (2024)
- [Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis](https://mlsanity.com/paper/2406.14678/) (2024)
- [Patterns of Lexical Ambiguity in Contextualised Language Models](https://mlsanity.com/paper/2109.13032/) (2021)
- [Representing Affect Information in Word Embeddings](https://mlsanity.com/paper/2209.10583/) (2022)
- [To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models](https://mlsanity.com/paper/2406.20054/) (2024)
- [Start Making Sense(s): A Developmental Probe of Attention Specialization Using Lexical Ambiguity](https://mlsanity.com/paper/2511.21974/) (2025)
- [Construction Grammar and Language Models](https://mlsanity.com/paper/2308.13315/) (2023)
- [Modelling Language using Large Language Models](https://mlsanity.com/paper/2404.09579/) (2024)
- [Prediction is not Explanation: Revisiting the Explanatory Capacity of Mapping Embeddings](https://mlsanity.com/paper/2508.13729/) (2025)
- [Transformers in the loop: Polarity in neural models of language](https://mlsanity.com/paper/2109.03926/) (2021)
- [Word-Embeddings Distinguish Denominal and Root-Derived Verbs in Semitic](https://mlsanity.com/paper/2208.05721/) (2022)
- [Translation from the Information Bottleneck Perspective: an Efficiency Analysis of Spatial Prepositions in Bitexts](https://mlsanity.com/paper/2603.19924/) (2026)

## Abstract (arXiv)

Understanding context-dependent variation in word meanings is a key aspect of human language comprehension supported by the lexicon. Lexicographic resources (e.g., WordNet) capture only some of this context-dependent variation; for example, they often do not encode how closely senses, or discretized word meanings, are related to one another. Our work investigates whether recent advances in NLP, specifically contextualized word embeddings, capture human-like distinctions between English word senses, such as polysemy and homonymy. We collect data from a behavioral, web-based experiment, in which participants provide judgments of the relatedness of multiple WordNet senses of a word in a two-dimensional spatial arrangement task. We find that participants' judgments of the relatedness between senses are correlated with distances between senses in the BERT embedding space. Homonymous senses (e.g., bat as mammal vs. bat as sports equipment) are reliably more distant from one another in the embedding space than polysemous ones (e.g., chicken as animal vs. chicken as meat). Our findings point towards the potential utility of continuous-space representations of sense meanings.