mlsanityData to 7 Oct 2026

Towards Understanding Linear Word Analogies

arXiv 1810.04882 ↗First posted 11 Oct 2018cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

A surprising property of word vectors is that word analogies can often be solved with vector arithmetic. However, it is unclear why arithmetic operators correspond to non-linear embedding models such as skip-gram with negative sampling (SGNS). We provide a formal explanation of this phenomenon without making the strong assumptions that past theories have made about the vector space and word distribution. Our theory has several implications. Past work has conjectured that linear substructures exist in vector spaces because relations can be represented as ratios; we prove that this holds for SGNS. We provide novel justification for the addition of SGNS word vectors by showing that it automatically down-weights the more frequent word, as weighting schemes do ad hoc. Lastly, we offer an information theoretic interpretation of Euclidean distance in vector spaces, justifying its use in capturing word dissimilarity.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 9; 2023: 13; 2024: 11; 2025: 14; 2026: 3.

57 papers.

Sort:

By area:

  1. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  2. 2023Getting aligned on representational alignmentcited by 142
  3. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  4. 2021BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?cited by 56
  5. 2022Problems with Cosine as a Measure of Embedding Similarity for High Frequency Wordscited by 49
  6. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  7. 2023PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learningcited by 31
  8. 2022Geodesic Multi-Modal Mixup for Robust Fine-Tuningcited by 29
  9. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  10. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoningcited by 28
  11. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  12. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
38 more
  1. 2022Multimodal Analogical Reasoning over Knowledge Graphscited by 21
  2. 2021Frequency-based Distortions in Contextualized Word Embeddingscited by 19
  3. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
  4. 2023netFound: Principled Design for Network Foundation Modelscited by 17
  5. 2023Uncovering Meanings of Embeddings via Partial Orthogonalitycited by 15
  6. 2024AAPL: Adding Attributes to Prompt Learning for Vision-Language Modelscited by 14
  7. 2024AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogiescited by 13
  8. 2022Large-scale Bilingual Language-Image Contrastive Learningcited by 12
  9. 2021Residual2Vec: Debiasing graph embedding with random graphscited by 11
  10. 2023Data-Efficient Multimodal Fusion on a Single GPUcited by 11
  11. 2026[b] = [d] - [t] + [p]: Self-supervised Speech Models Discover Phonological Vector Arithmeticcited by 11
  12. 2025On the Emergence of Linear Analogies in Word Embeddingscited by 9
  13. 2021The Rediscovery Hypothesis: Language Models Need to Meet Linguisticscited by 7
  14. 2022Richer Countries and Richer Representationscited by 7
  15. 2021Theoretical foundations and limits of word embeddings: what types of meaning can they capture?cited by 5
  16. 2023Solving Cosine Similarity Underestimation between High Frequency Words by L2 Norm Discountingcited by 5
  17. 2023Bridging Continuous and Discrete Spaces: Interpretable Sentence Representation Learning via Compositional Operationscited by 5
  18. 2021Revisiting Additive Compositionality: AND, OR and NOT Operations with Word Embeddingscited by 4
  19. 2022Stock Embeddings: Learning Distributed Representations for Financial Assetscited by 4
  20. 2024Statistical Uncertainty in Word Embeddings: GloVe-Vcited by 4
  21. 2025Eta-WavLM: Efficient Speaker Identity Removal in Self-Supervised Speech Representations Using a Simple Linear Equationcited by 3
  22. 2021Paraphrases do not explain word analogiescited by 2
  23. 2023Contrastive Loss is All You Need to Recover Analogies as Parallel Linescited by 2
  24. 2024Generating Human Understandable Explanations for Node Embeddingscited by 2
  25. 2024Can Multimodal Large Language Model Think Analogically?cited by 2
  26. 2025Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representationscited by 2
  27. 2025FORGE: Foundational Optimization Representations from Graph Embeddingscited by 2
  28. 2025Testing the assumptions about the geometry of sentence embedding spaces: the cosine measure need not applycited by 2
  29. 2025Probability Signature: Bridging Data Semantics and Embedding Structure in Language Modelscited by 2
  30. 2025Demystifying Network Foundation Modelscited by 2
  31. 2023Simulating News Recommendation Ecosystem for Fun and Profitcited by 1
  32. 2023Bidirectional Attention as a Mixture of Continuous Word Expertscited by 1
  33. 2023RelBERT: Embedding Relations with Language Modelscited by 1
  34. 2024Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNetscited by 1
  35. 2025Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Producecited by 1
  36. 2025How do Transformer Embeddings Represent Compositions? A Functional Analysiscited by 1
  37. 2025Intervening to Learn and Compose Causally Disentangled Representationscited by 1
  38. 2025Vocabulary embeddings organize linguistic structure early in language model trainingcited by 1

50 most cited of 57 All 57 (JSON)