# Analogies Explained: Towards Understanding Word Embeddings

arXiv: https://arxiv.org/abs/1901.09813 · HTML: https://mlsanity.com/paper/1901.09813/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2019.

## Built on by (most cited first, 30 of 64)

- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [Word Embeddings Are Steers for Language Models](https://mlsanity.com/paper/2305.12798/) (2023)
- [Concept Algebra for (Score-Based) Text-Controlled Generative Models](https://mlsanity.com/paper/2302.03693/) (2023)
- [BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?](https://mlsanity.com/paper/2105.04949/) (2021)
- [Linear Spaces of Meanings: Compositional Structures in Vision-Language Models](https://mlsanity.com/paper/2302.14383/) (2023)
- [On the Explainability of Natural Language Processing Deep Models](https://mlsanity.com/paper/2210.06929/) (2022)
- [Cross-Entropy Is All You Need To Invert the Data Generating Process](https://mlsanity.com/paper/2410.21869/) (2024)
- [Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Models](https://mlsanity.com/paper/2402.09236/) (2024)
- [Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations](https://mlsanity.com/paper/2408.15417/) (2024)
- [VASR: Visual Analogies of Situation Recognition](https://mlsanity.com/paper/2212.04542/) (2022)
- [Norm of Word Embedding Encodes Information Gain](https://mlsanity.com/paper/2212.09663/) (2022)
- [PaCE: Parsimonious Concept Engineering for Large Language Models](https://mlsanity.com/paper/2406.04331/) (2024)
- [Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference](https://mlsanity.com/paper/2403.04082/) (2024)
- [Uncovering Meanings of Embeddings via Partial Orthogonality](https://mlsanity.com/paper/2310.17611/) (2023)
- [All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling](https://mlsanity.com/paper/2410.23501/) (2024)
- [On the Emergence of Linear Analogies in Word Embeddings](https://mlsanity.com/paper/2505.18651/) (2025)
- [ProTIP: Progressive Tool Retrieval Improves Planning](https://mlsanity.com/paper/2312.10332/) (2023)
- [Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abduction](https://mlsanity.com/paper/2305.12660/) (2023)
- [Linguistic Dependencies and Statistical Dependence](https://mlsanity.com/paper/2104.08685/) (2021)
- [From Data Statistics to Feature Geometry: How Correlations Shape Superposition](https://mlsanity.com/paper/2603.09972/) (2026)
- [The Rediscovery Hypothesis: Language Models Need to Meet Linguistics](https://mlsanity.com/paper/2103.01819/) (2021)
- [On the Properties and Estimation of Pointwise Mutual Information Profiles](https://mlsanity.com/paper/2310.10240/) (2023)
- [Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation](https://mlsanity.com/paper/2104.04916/) (2021)
- [Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey](https://mlsanity.com/paper/2204.08226/) (2022)
- [Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective](https://mlsanity.com/paper/2204.12316/) (2022)
- [A Probabilistic Model Behind Self-Supervised Learning](https://mlsanity.com/paper/2402.01399/) (2024)
- [Theoretical Understandings of Product Embedding for E-commerce Machine Learning](https://mlsanity.com/paper/2102.12029/) (2021)
- [Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddings](https://mlsanity.com/paper/2105.08585/) (2021)

## Abstract (arXiv)

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing that we re-interpret as word transformation, a mathematical description of "wx is to wy". From these concepts we prove existence of linear relationships between W2V-type embeddings that underlie the analogical phenomenon, identifying explicit error terms.