# Probing for Semantic Classes: Diagnosing the Meaning Content of Word Embeddings

arXiv: https://arxiv.org/abs/1906.03608 · HTML: https://mlsanity.com/paper/1906.03608/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jun 2019.

## Built on by (most cited first, 15 of 15)

- [A Closer Look at How Fine-tuning Changes BERT](https://mlsanity.com/paper/2106.14282/) (2021)
- [Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages](https://mlsanity.com/paper/2203.14139/) (2022)
- [DirectProbe: Studying Representations without Classifiers](https://mlsanity.com/paper/2104.05904/) (2021)
- [The Life Cycle of Knowledge in Big Language Models: A Survey](https://mlsanity.com/paper/2303.07616/) (2023)
- [Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case](https://mlsanity.com/paper/2102.11115/) (2021)
- [Inspecting the concept knowledge graph encoded by modern language models](https://mlsanity.com/paper/2105.13471/) (2021)
- [LLaMAs Have Feelings Too: Unveiling Sentiment and Emotion Representations in LLaMA Models Through Probing](https://mlsanity.com/paper/2505.16491/) (2025)
- [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023)
- [Improving Entity Linking through Semantic Reinforced Entity Embeddings](https://mlsanity.com/paper/2106.08495/) (2021)
- [Transformers, Contextualism, and Polysemy](https://mlsanity.com/paper/2404.09577/) (2024)
- [Sentence Ambiguity, Grammaticality and Complexity Probes](https://mlsanity.com/paper/2210.06928/) (2022)
- [Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?](https://mlsanity.com/paper/2405.07099/) (2024)
- [DeepCodeProbe: Towards Understanding What Models Trained on Code Learn](https://mlsanity.com/paper/2407.08890/) (2024)
- [GausSetExpander: A Simple Approach for Entity Set Expansion](https://mlsanity.com/paper/2202.13649/) (2022)
- [Probing of Quantitative Values in Abstractive Summarization Models](https://mlsanity.com/paper/2210.00667/) (2022)

## Abstract (arXiv)

Word embeddings typically represent different meanings of a word in a single conflated vector. Empirical analysis of embeddings of ambiguous words is currently limited by the small size of manually annotated resources and by the fact that word senses are treated as unrelated individual concepts. We present a large dataset based on manual Wikipedia annotations and word senses, where word senses from different words are related by semantic classes. This is the basis for novel diagnostic tests for an embedding's content: we probe word embeddings for semantic classes and analyze the embedding space by classifying embeddings into semantic classes. Our main findings are: (i) Information about a sense is generally represented well in a single-vector embedding - if the sense is frequent. (ii) A classifier can accurately predict whether a word is single-sense or multi-sense, based only on its embedding. (iii) Although rare senses are not well represented in single-vector embeddings, this does not have negative impact on an NLP application whose performance depends on frequent senses.