# What do you mean, BERT? Assessing BERT as a Distributional Semantics Model

arXiv: https://arxiv.org/abs/1911.05758 · HTML: https://mlsanity.com/paper/1911.05758/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2019.

## Built on by (most cited first, 14 of 14)

- [IsoScore: Measuring the Uniformity of Embedding Space Utilization](https://mlsanity.com/paper/2108.07344/) (2021)
- [What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Code](https://mlsanity.com/paper/2202.06840/) (2022)
- [A comparative evaluation and analysis of three generations of Distributional Semantic Models](https://mlsanity.com/paper/2105.09825/) (2021)
- [Obtaining Better Static Word Embeddings Using Contextual Embedding Models](https://mlsanity.com/paper/2106.04302/) (2021)
- [LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond](https://mlsanity.com/paper/2105.12449/) (2021)
- [The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERT](https://mlsanity.com/paper/2101.09115/) (2021)
- [Stable Anisotropic Regularization](https://mlsanity.com/paper/2305.19358/) (2023)
- [Semantics of Multiword Expressions in Transformer-Based Models: A Survey](https://mlsanity.com/paper/2401.15393/) (2024)
- [A Psycholinguistic Analysis of BERT's Representations of Compounds](https://mlsanity.com/paper/2302.07232/) (2023)
- [Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physics](https://mlsanity.com/paper/2411.14877/) (2024)
- [Using Distributional Principles for the Semantic Study of Contextual Language Models](https://mlsanity.com/paper/2111.12174/) (2021)
- [Do Transformers know symbolic rules, and would we know if they did?](https://mlsanity.com/paper/2203.00162/) (2022)
- [Agentivit`a e telicit`a in GilBERTo: implicazioni cognitive](https://mlsanity.com/paper/2307.02910/) (2023)
- [Learning grounded word meaning representations on similarity graphs](https://mlsanity.com/paper/2109.03084/) (2021)

## Abstract (arXiv)

Contextualized word embeddings, i.e. vector representations for words in context, are naturally seen as an extension of previous noncontextual distributional semantic models. In this work, we focus on BERT, a deep neural network that produces contextualized embeddings and has set the state-of-the-art in several semantic tasks, and study the semantic coherence of its embedding space. While showing a tendency towards coherence, BERT does not fully live up to the natural expectations for a semantic vector space. In particular, we find that the position of the sentence in which a word occurs, while having no meaning correlates, leaves a noticeable trace on the word embeddings and disturbs similarity relationships.