mlsanityData to 7 Oct 2026

Analogies Explained: Towards Understanding Word Embeddings

arXiv 1901.09813 ↗First posted 28 Jan 2019cs.CL cs.LG stat.MLcited by 63

Not on the map. Why?

Abstract (arXiv)

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing that we re-interpret as word transformation, a mathematical description of "wx is to wy". From these concepts we prove existence of linear relationships between W2V-type embeddings that underlie the analogical phenomenon, identifying explicit error terms.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 16; 2024: 19; 2025: 7; 2026: 4.

63 papers.

Sort:

By area:

  1. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  2. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  3. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  4. 2023Word Embeddings Are Steers for Language Modelscited by 70
  5. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59
  6. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  7. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  8. 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
  9. 2024Cross-Entropy Is All You Need To Invert the Data Generating Processcited by 30
  10. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  11. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  12. 2022VASR: Visual Analogies of Situation Recognitioncited by 22
38 more
  1. 2022Norm of Word Embedding Encodes Information Gaincited by 21
  2. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  3. 2023Uncovering Meanings of Embeddings via Partial Orthogonalitycited by 15
  4. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  5. 2024All or None: Identifiable Linear Properties of Next-token Predictors in Language Modelingcited by 14
  6. 2023ProTIP: Progressive Tool Retrieval Improves Planningcited by 12
  7. 2023Beneath Surface Similarity: Large Language Models Make Reasonable Scientific Analogies after Structure Abductioncited by 11
  8. 2021Linguistic Dependencies and Statistical Dependencecited by 10
  9. 2025On the Emergence of Linear Analogies in Word Embeddingscited by 9
  10. 2026From Data Statistics to Feature Geometry: How Correlations Shape Superpositioncited by 9
  11. 2021The Rediscovery Hypothesis: Language Models Need to Meet Linguisticscited by 7
  12. 2023On the Properties and Estimation of Pointwise Mutual Information Profilescited by 7
  13. 2021Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisationcited by 6
  14. 2022Empirical Evaluation and Theoretical Analysis for Representation Learning: A Surveycited by 5
  15. 2022Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspectivecited by 5
  16. 2024A Probabilistic Model Behind Self-Supervised Learningcited by 5
  17. 2021Theoretical Understandings of Product Embedding for E-commerce Machine Learningcited by 4
  18. 2021Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddingscited by 4
  19. 2022On the Complexity of Bayesian Generalizationcited by 3
  20. 2023FAME: Flexible, Scalable Analogy Mappings Enginecited by 3
  21. 2024Boosting Scientific Concepts Understanding: Can Analogy from Teacher Models Empower Student Models?cited by 3
  22. 2021Paraphrases do not explain word analogiescited by 2
  23. 2023Contrastive Loss is All You Need to Recover Analogies as Parallel Linescited by 2
  24. 2023Why bother with geometry? On the relevance of linear decompositions of Transformer embeddingscited by 2
  25. 2024Generating Human Understandable Explanations for Node Embeddingscited by 2
  26. 2024Any four real numbers are on all fours with analogycited by 2
  27. 2025Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representationscited by 2
  28. 2021Compositional Clustering: Applications to Multi-Label Object Recognition and Speaker Identificationcited by 1
  29. 2022Style-Content Disentanglement in Language-Image Pretraining Representations for Zero-Shot Sketch-to-Image Synthesiscited by 1
  30. 2023Pretrained Embeddings for E-commerce Machine Learning: When it Fails and Why?cited by 1
  31. 2023Simulating News Recommendation Ecosystem for Fun and Profitcited by 1
  32. 2023Bidirectional Attention as a Mixture of Continuous Word Expertscited by 1
  33. 2023RelBERT: Embedding Relations with Language Modelscited by 1
  34. 2024Predicting drug-gene relations via analogy tasks with word embeddingscited by 1
  35. 2024Compositional Structures in Neural Embedding and Interaction Decompositionscited by 1
  36. 2024KnowledgePrompts: Exploring the Abilities of Large Language Models to Solve Proportional Analogies via Knowledge-Enhanced Promptingcited by 1
  37. 2025Intervening to Learn and Compose Causally Disentangled Representationscited by 1
  38. 2025Vocabulary embeddings organize linguistic structure early in language model trainingcited by 1

50 most cited of 63 All 63 (JSON)