# Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings

arXiv: https://arxiv.org/abs/1909.10430 · HTML: https://mlsanity.com/paper/1909.10430/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Sep 2019.

## Built on by (most cited first, 30 of 56)

- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Vision-Language Models Provide Promptable Representations for Reinforcement Learning](https://mlsanity.com/paper/2402.02651/) (2024)
- [Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Senses](https://mlsanity.com/paper/2104.14694/) (2021)
- [Exploring the Role of BERT Token Representations to Explain Sentence Probing Results](https://mlsanity.com/paper/2104.01477/) (2021)
- [Characterizing English Variation across Social Media Communities with BERT](https://mlsanity.com/paper/2102.06820/) (2021)
- [How to Dissect a Muppet: The Structure of Transformer Embedding Spaces](https://mlsanity.com/paper/2206.03529/) (2022)
- [A comparative evaluation and analysis of three generations of Distributional Semantic Models](https://mlsanity.com/paper/2105.09825/) (2021)
- [MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models](https://mlsanity.com/paper/2109.09237/) (2021)
- [The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT](https://mlsanity.com/paper/2101.09115/) (2021)
- [Inspecting the concept knowledge graph encoded by modern language models](https://mlsanity.com/paper/2105.13471/) (2021)
- [LatentLens: Revealing Highly Interpretable Visual Tokens in LLMs](https://mlsanity.com/paper/2602.00462/) (2026)
- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)
- [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [From Directions to Regions: Decomposing Activations in Language Models via Local Geometry](https://mlsanity.com/paper/2602.02464/) (2026)
- [RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English)](https://mlsanity.com/paper/2105.13266/) (2021)
- [Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy](https://mlsanity.com/paper/2106.13553/) (2021)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)
- [Unsupervised Lexical Substitution with Decontextualised Embeddings](https://mlsanity.com/paper/2209.08236/) (2022)
- [Incorporating Word Sense Disambiguation in Neural Language Models](https://mlsanity.com/paper/2106.07967/) (2021)
- [The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations](https://mlsanity.com/paper/2402.14616/) (2024)
- [Embedding Recycling for Language Models](https://mlsanity.com/paper/2207.04993/) (2022)
- [generAItor: Tree-in-the-Loop Text Generation for Language Model Explainability and Adaptation](https://mlsanity.com/paper/2403.07627/) (2024)
- [Transformers, Contextualism, and Polysemy](https://mlsanity.com/paper/2404.09577/) (2024)
- [Patterns of Lexical Ambiguity in Contextualised Language Models](https://mlsanity.com/paper/2109.13032/) (2021)
- [Neural Sequence-to-Sequence Modeling with Attention by Leveraging Deep Learning Architectures for Enhanced Contextual Understanding in Abstractive Text Summarization](https://mlsanity.com/paper/2404.08685/) (2024)
- [Using Paraphrases to Study Properties of Contextual Embeddings](https://mlsanity.com/paper/2207.05553/) (2022)
- [Representing Affect Information in Word Embeddings](https://mlsanity.com/paper/2209.10583/) (2022)
- [Word sense extension](https://mlsanity.com/paper/2306.05609/) (2023)
- [Finding Challenging Metaphors that Confuse Pretrained Language Models](https://mlsanity.com/paper/2401.16012/) (2024)

## Abstract (arXiv)

Contextualized word embeddings (CWE) such as provided by ELMo (Peters et al., 2018), Flair NLP (Akbik et al., 2018), or BERT (Devlin et al., 2019) are a major recent innovation in NLP. CWEs provide semantic vector representations of words depending on their respective context. Their advantage over static word embeddings has been shown for a number of tasks, such as text classification, sequence tagging, or machine translation. Since vectors of the same word type can vary depending on the respective context, they implicitly provide a model for word sense disambiguation (WSD). We introduce a simple but effective approach to WSD using a nearest neighbor classification on CWEs. We compare the performance of different CWE models for the task and can report improvements above the current state of the art for two standard WSD benchmark datasets. We further show that the pre-trained BERT model is able to place polysemic words into distinct 'sense' regions of the embedding space, while ELMo and Flair NLP do not seem to possess this ability.