# What the Vec? Towards Probabilistically Grounded Embeddings

arXiv: https://arxiv.org/abs/1805.12164 · HTML: https://mlsanity.com/paper/1805.12164/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 May 2018.

## Built on by (most cited first, 8 of 8)

- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey](https://mlsanity.com/paper/2204.08226/) (2022)
- [Theoretical Understandings of Product Embedding for E-commerce Machine Learning](https://mlsanity.com/paper/2102.12029/) (2021)
- [Language Models as Zero-shot Visual Semantic Learners](https://mlsanity.com/paper/2107.12021/) (2021)
- [Probabilistic Embeddings with Laplacian Graph Priors](https://mlsanity.com/paper/2204.01846/) (2022)
- [From Hyperbolic Geometry Back to Word Embeddings](https://mlsanity.com/paper/2204.12481/) (2022)
- [Contrastive Learning as Kernel Approximation](https://mlsanity.com/paper/2309.02651/) (2023)
- [The cell as a token: high-dimensional geometry in language models and cell embeddings](https://mlsanity.com/paper/2503.20278/) (2025)

## Abstract (arXiv)

Word2Vec (W2V) and GloVe are popular, fast and efficient word embedding algorithms. Their embeddings are widely used and perform well on a variety of natural language processing tasks. Moreover, W2V has recently been adopted in the field of graph embedding, where it underpins several leading algorithms. However, despite their ubiquity and relatively simple model architecture, a theoretical understanding of what the embedding parameters of W2V and GloVe learn and why that is useful in downstream tasks has been lacking. We show that different interactions between PMI vectors reflect semantic word relationships, such as similarity and paraphrasing, that are encoded in low dimensional word embeddings under a suitable projection, theoretically explaining why embeddings of W2V and GloVe work. As a consequence, we also reveal an interesting mathematical interconnection between the considered semantic relationships themselves.