# The Spectral Underpinning of word2vec

arXiv: https://arxiv.org/abs/2002.12317 · HTML: https://mlsanity.com/paper/2002.12317/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Feb 2020.

## Built on by (most cited first, 2 of 2)

- [Spectral Top-Down Recovery of Latent Tree Models](https://mlsanity.com/paper/2102.13276/) (2021)
- [Learning distributed representations with efficient SoftMax normalization](https://mlsanity.com/paper/2303.17475/) (2023)

## Abstract (arXiv)

word2vec due to Mikolov textitet al. (2013) is a word embedding method that is widely used in natural language processing. Despite its great success and frequent use, theoretical justification is still lacking. The main contribution of our paper is to propose a rigorous analysis of the highly nonlinear functional of word2vec. Our results suggest that word2vec may be primarily driven by an underlying spectral method. This insight may open the door to obtaining provable guarantees for word2vec. We support these findings by numerical simulations. One fascinating open question is whether the nonlinear properties of word2vec that are not captured by the spectral method are beneficial and, if so, by what mechanism.