mlsanityData to 7 Oct 2026

What do you mean, BERT? Assessing BERT as a Distributional Semantics Model

arXiv 1911.05758 ↗First posted 13 Nov 2019cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Contextualized word embeddings, i.e. vector representations for words in context, are naturally seen as an extension of previous noncontextual distributional semantic models. In this work, we focus on BERT, a deep neural network that produces contextualized embeddings and has set the state-of-the-art in several semantic tasks, and study the semantic coherence of its embedding space. While showing a tendency towards coherence, BERT does not fully live up to the natural expectations for a semantic vector space. In particular, we find that the position of the sentence in which a word occurs, while having no meaning correlates, leaves a noticeable trace on the word embeddings and disturbs similarity relationships.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 2; 2023: 3; 2024: 2.

14 papers.

Sort:

  1. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 52
  2. 2022What Do They Capture? -- A Structural Analysis of Pre-Trained Language Models for Source Codecited by 50
  3. 2021A comparative evaluation and analysis of three generations of Distributional Semantic Modelscited by 19
  4. 2021Obtaining Better Static Word Embeddings Using Contextual Embedding Modelscited by 18
  5. 2021LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyondcited by 17
  6. 2021The heads hypothesis: A unifying statistical approach towards understanding multi-headed attention in BERTcited by 15
  7. 2023Stable Anisotropic Regularizationcited by 12
  8. 2024Semantics of Multiword Expressions in Transformer-Based Models: A Surveycited by 9
  9. 2023A Psycholinguistic Analysis of BERT's Representations of Compoundscited by 7
  10. 2024Astro-HEP-BERT: A bidirectional language model for studying the meanings of concepts in astrophysics and high energy physicscited by 2
  11. 2021Using Distributional Principles for the Semantic Study of Contextual Language Modelscited by 1
  12. 2022Do Transformers know symbolic rules, and would we know if they did?cited by 1
2 more
  1. 2023Agentivit`a e telicit`a in GilBERTo: implicazioni cognitivecited by 1
  2. 2021Learning grounded word meaning representations on similarity graphscited by 0