# Shape of Elephant: Study of Macro Properties of Word Embeddings Spaces

arXiv: https://arxiv.org/abs/2106.06964 · HTML: https://mlsanity.com/paper/2106.06964/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 11 references that are in the map.
- First posted on arXiv 13 Jun 2021.

## Builds on (most cited first, 5 of 5)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [Intrinsic Probing through Dimension Selection](https://mlsanity.com/paper/2010.02812/) (2020)
- [Stolen Probability: A Structural Weakness of Neural Language Models](https://mlsanity.com/paper/2005.02433/) (2020)
- [Semantic Structure and Interpretability of Word Embeddings](https://mlsanity.com/paper/1711.00331/) (2017)

## Abstract (arXiv)

Pre-trained word representations became a key component in many NLP tasks. However, the global geometry of the word embeddings remains poorly understood. In this paper, we demonstrate that a typical word embeddings cloud is shaped as a high-dimensional simplex with interpretable vertices and propose a simple yet effective method for enumeration of these vertices. We show that the proposed method can detect and describe vertices of the simplex for GloVe and fasttext spaces.