# Issues in evaluating semantic spaces using word analogies

arXiv: https://arxiv.org/abs/1606.07736 · HTML: https://mlsanity.com/paper/1606.07736/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jun 2016.

## Built on by (most cited first, 27 of 27)

- [BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?](https://mlsanity.com/paper/2105.04949/) (2021)
- [Linear Spaces of Meanings: Compositional Structures in Vision-Language Models](https://mlsanity.com/paper/2302.14383/) (2023)
- [Life is a Circus and We are the Clowns: Automatically Finding Analogies between Situations and Processes](https://mlsanity.com/paper/2210.12197/) (2022)
- [In-Context Analogical Reasoning with Pre-Trained Language Models](https://mlsanity.com/paper/2305.17626/) (2023)
- [Language-Specific Latent Process Hinders Cross-Lingual Performance](https://mlsanity.com/paper/2505.13141/) (2025)
- [A comparative evaluation and analysis of three generations of Distributional Semantic Models](https://mlsanity.com/paper/2105.09825/) (2021)
- [Idiosyncratic but not Arbitrary: Learning Idiolects in Online Registers Reveals Distinctive yet Consistent Individual Styles](https://mlsanity.com/paper/2109.03158/) (2021)
- [Understanding polysemanticity in neural networks through coding theory](https://mlsanity.com/paper/2401.17975/) (2024)
- [Distilling Relation Embeddings from Pre-trained Language Models](https://mlsanity.com/paper/2110.15705/) (2021)
- [Probabilistic Analogical Mapping with Semantic Relation Networks](https://mlsanity.com/paper/2103.16704/) (2021)
- [Revisiting Word Embeddings in the LLM Era](https://mlsanity.com/paper/2402.11094/) (2024)
- [Can language models learn analogical reasoning? Investigating training objectives and comparisons to human performance](https://mlsanity.com/paper/2310.05597/) (2023)
- [ParallelPARC: A Scalable Pipeline for Generating Natural-Language Analogies](https://mlsanity.com/paper/2403.01139/) (2024)
- [Evaluating the Construct Validity of Text Embeddings with Application to Survey Questions](https://mlsanity.com/paper/2202.09166/) (2022)
- [Construction Artifacts in Metaphor Identification Datasets](https://mlsanity.com/paper/2311.00790/) (2023)
- [Statistical Uncertainty in Word Embeddings: GloVe-V](https://mlsanity.com/paper/2406.12165/) (2024)
- [Paraphrases do not explain word analogies](https://mlsanity.com/paper/2102.11749/) (2021)
- [Semantic properties of English nominal pluralization: Insights from word embeddings](https://mlsanity.com/paper/2203.15424/) (2022)
- [Contrastive Loss is All You Need to Recover Analogies as Parallel Lines](https://mlsanity.com/paper/2306.08221/) (2023)
- [Any four real numbers are on all fours with analogy](https://mlsanity.com/paper/2407.18770/) (2024)
- [RelBERT: Embedding Relations with Language Models](https://mlsanity.com/paper/2310.00299/) (2023)
- [Beer2Vec : Extracting Flavors from Reviews for Thirst-Quenching Recommandations](https://mlsanity.com/paper/2208.04223/) (2022)
- [On the Information Content of Predictions in Word Analogy Tests](https://mlsanity.com/paper/2210.09972/) (2022)
- [Revisiting Word Embeddings in the LLM Era](https://mlsanity.com/paper/2502.19607/) (2025)
- [Large Language Models provide support for the parallelogram theory of analogy](https://mlsanity.com/paper/2603.19066/) (2026)
- [A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability](https://mlsanity.com/paper/2608.29034/) (2026)
- [Vision Transformers Learn Word Meaning from Images Alone](https://mlsanity.com/paper/2610.13252/) (2026)

## Abstract (arXiv)

The offset method for solving word analogies has become a standard evaluation tool for vector-space semantic models: it is considered desirable for a space to represent semantic relations as consistent vector offsets. We show that the method's reliance on cosine similarity conflates offset consistency with largely irrelevant neighborhood structure, and propose simple baselines that should be used to improve the utility of the method in vector space evaluation.