# Contrastive Loss is All You Need to Recover Analogies as Parallel Lines

arXiv: https://arxiv.org/abs/2306.08221 · HTML: https://mlsanity.com/paper/2306.08221/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 14 Jun 2023.

## Builds on (most cited first, 7 of 7)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [Analogies Explained: Towards Understanding Word Embeddings](https://mlsanity.com/paper/1901.09813/) (2019)
- [Towards Understanding Linear Word Analogies](https://mlsanity.com/paper/1810.04882/) (2018)
- [Issues in evaluating semantic spaces using word analogies](https://mlsanity.com/paper/1606.07736/) (2016)
- [Analogies minus analogy test: measuring regularities in word embeddings](https://mlsanity.com/paper/2010.03446/) (2020)
- [Paraphrases do not explain word analogies](https://mlsanity.com/paper/2102.11749/) (2021)

## Built on by (most cited first, 2 of 2)

- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)

## Abstract (arXiv)

While static word embedding models are known to represent linguistic analogies as parallel lines in high-dimensional space, the underlying mechanism as to why they result in such geometric structures remains obscure. We find that an elementary contrastive-style method employed over distributional information performs competitively with popular word embedding models on analogy recovery tasks, while achieving dramatic speedups in training time. Further, we demonstrate that a contrastive loss is sufficient to create these parallel structures in word embeddings, and establish a precise relationship between the co-occurrence statistics and the geometric structure of the resulting word embeddings.