# Probing Taxonomic and Thematic Embeddings for Taxonomic Information

arXiv: https://arxiv.org/abs/2301.10656 · HTML: https://mlsanity.com/paper/2301.10656/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 68 references that are in the map.
- First posted on arXiv 25 Jan 2023.

## Builds on (most cited first, 16 of 16)

- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Analysis Methods in Neural Language Processing: A Survey](https://mlsanity.com/paper/1812.08951/) (2018)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Analyzing Individual Neurons in Pre-trained Language Models](https://mlsanity.com/paper/2010.02695/) (2020)
- [Factors Influencing the Surprising Instability of Word Embeddings](https://mlsanity.com/paper/1804.09692/) (2018)
- [Improving Hypernymy Detection with an Integrated Path-based and Distributional Method](https://mlsanity.com/paper/1603.06076/) (2016)
- [HyperLex: A Large-Scale Evaluation of Graded Lexical Entailment](https://mlsanity.com/paper/1608.02117/) (2016)
- [Hearst Patterns Revisited: Automatic Hypernym Detection from Large Text Corpora](https://mlsanity.com/paper/1806.03191/) (2018)
- [Discovering Salient Neurons in Deep NLP Models](https://mlsanity.com/paper/2206.13288/) (2022)
- [Hierarchical Embeddings for Hypernymy Detection and Directionality](https://mlsanity.com/paper/1707.07273/) (2017)
- [Hypernyms under Siege: Linguistically-motivated Artillery for Hypernymy Detection](https://mlsanity.com/paper/1612.04460/) (2016)
- [Nine Features in a Random Forest to Learn Taxonomical Semantic Relations](https://mlsanity.com/paper/1603.08702/) (2016)
- [Predicting Semantic Relations using Global Graph Properties](https://mlsanity.com/paper/1808.08644/) (2018)
- [Probing with Noise: Unpicking the Warp and Weft of Embeddings](https://mlsanity.com/paper/2210.12206/) (2022)

## Built on by (most cited first, 1 of 1)

- [Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space](https://mlsanity.com/paper/2304.14333/) (2023)

## Abstract (arXiv)

Modelling taxonomic and thematic relatedness is important for building AI with comprehensive natural language understanding. The goal of this paper is to learn more about how taxonomic information is structurally encoded in embeddings. To do this, we design a new hypernym-hyponym probing task and perform a comparative probing study of taxonomic and thematic SGNS and GloVe embeddings. Our experiments indicate that both types of embeddings encode some taxonomic information, but the amount, as well as the geometric properties of the encodings, are independently related to both the encoder architecture, as well as the embedding training data. Specifically, we find that only taxonomic embeddings carry taxonomic information in their norm, which is determined by the underlying distribution in the data.