# Deconstructing word embedding algorithms

arXiv: https://arxiv.org/abs/2011.07013 · HTML: https://mlsanity.com/paper/2011.07013/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2020.

## Built on by (most cited first, 5 of 5)

- [EigenNoise: A Contrastive Prior to Warm-Start Representations](https://mlsanity.com/paper/2205.04376/) (2022)
- [ALANNO: An Active Learning Annotation System for Mortals](https://mlsanity.com/paper/2211.06224/) (2022)
- [To Know by the Company Words Keep and What Else Lies in the Vicinity](https://mlsanity.com/paper/2205.00148/) (2022)
- [Contrastive Learning as Kernel Approximation](https://mlsanity.com/paper/2309.02651/) (2023)
- [Is neural semantic parsing good at ellipsis resolution, or isn't it?](https://mlsanity.com/paper/2508.00121/) (2025)

## Abstract (arXiv)

Word embeddings are reliable feature representations of words used to obtain high quality results for various NLP applications. Uncontextualized word embeddings are used in many NLP tasks today, especially in resource-limited settings where high memory capacity and GPUs are not available. Given the historical success of word embeddings in NLP, we propose a retrospective on some of the most well-known word embedding algorithms. In this work, we deconstruct Word2vec, GloVe, and others, into a common form, unveiling some of the common conditions that seem to be required for making performant word embeddings. We believe that the theoretical findings in this paper can provide a basis for more informed development of future models.