# Understanding Composition of Word Embeddings via Tensor Decomposition

arXiv: https://arxiv.org/abs/1902.00613 · HTML: https://mlsanity.com/paper/1902.00613/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2019.

## Built on by (most cited first, 4 of 4)

- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [N-gram Prediction and Word Difference Representations for Language Modeling](https://mlsanity.com/paper/2409.03295/) (2024)
- [Sentence Curve Language Models](https://mlsanity.com/paper/2602.01807/) (2026)

## Abstract (arXiv)

Word embedding is a powerful tool in natural language processing. In this paper we consider the problem of word embedding composition --- given vector representations of two words, compute a vector for the entire phrase. We give a generative model that can capture specific syntactic relations between words. Under our model, we prove that the correlations between three words (measured by their PMI) form a tensor that has an approximate low rank Tucker decomposition. The result of the Tucker decomposition gives the word embeddings as well as a core tensor, which can be used to produce better compositions of the word embeddings. We also complement our theoretical results with experiments that verify our assumptions, and demonstrate the effectiveness of the new composition method.