# Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

arXiv: https://arxiv.org/abs/1605.02276 · HTML: https://mlsanity.com/paper/1605.02276/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 May 2016.

## Built on by (most cited first, 25 of 25)

- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Extrinsic Evaluation of Cultural Competence in Large Language Models](https://mlsanity.com/paper/2406.11565/) (2024)
- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [Obtaining Better Static Word Embeddings Using Contextual Embedding Models](https://mlsanity.com/paper/2106.04302/) (2021)
- [Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost](https://mlsanity.com/paper/2203.07860/) (2022)
- [Multi-sense embeddings through a word sense disambiguation process](https://mlsanity.com/paper/2101.08700/) (2021)
- [Enhancing Interpretability using Human Similarity Judgements to Prune Word Embeddings](https://mlsanity.com/paper/2310.10262/) (2023)
- [Theoretical foundations and limits of word embeddings: what types of meaning can they capture?](https://mlsanity.com/paper/2107.10413/) (2021)
- [Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddings](https://mlsanity.com/paper/2204.12386/) (2022)
- [No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Media](https://mlsanity.com/paper/2211.03634/) (2022)
- [Fine-Tuned LLMs are "Time Capsules" for Tracking Societal Bias Through Books](https://mlsanity.com/paper/2502.05331/) (2025)
- [It's AI Match: A Two-Step Approach for Schema Matching Using Embeddings](https://mlsanity.com/paper/2203.04366/) (2022)
- [Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitions](https://mlsanity.com/paper/2305.07303/) (2023)
- [Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spaces](https://mlsanity.com/paper/2310.15905/) (2023)
- [A Comprehensive Analysis of Static Word Embeddings for Turkish](https://mlsanity.com/paper/2405.07778/) (2024)
- [SuperSim: a test set for word similarity and relatedness in Swedish](https://mlsanity.com/paper/2104.05228/) (2021)
- [Semantic Certainty Assessment in Vector Retrieval Systems: A Novel Framework for Embedding Quality Evaluation](https://mlsanity.com/paper/2507.05933/) (2025)
- [Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention](https://mlsanity.com/paper/2510.01652/) (2025)
- [Understanding and Enhancing the Use of Context for Machine Translation](https://mlsanity.com/paper/2102.10437/) (2021)
- [An empirical analysis of phrase-based and neural machine translation](https://mlsanity.com/paper/2103.03108/) (2021)
- [Term Interrelations and Trends in Software Engineering](https://mlsanity.com/paper/2108.09529/) (2021)
- [On the Information Content of Predictions in Word Analogy Tests](https://mlsanity.com/paper/2210.09972/) (2022)
- [Word Embedding Dimension Reduction via Weakly-Supervised Feature Selection](https://mlsanity.com/paper/2407.12342/) (2024)
- [Interpretable Syntactic Representations Enable Hierarchical Word Vectors](https://mlsanity.com/paper/2411.08384/) (2024)
- [Set-Theoretic Compositionality of Sentence Embeddings](https://mlsanity.com/paper/2502.20975/) (2025)

## Abstract (arXiv)

Lacking standardized extrinsic evaluation methods for vector representations of words, the NLP community has relied heavily on word similarity tasks as a proxy for intrinsic evaluation of word vectors. Word similarity evaluation, which correlates the distance between vectors and human judgments of semantic similarity is attractive, because it is computationally inexpensive and fast. In this paper we present several problems associated with the evaluation of word vectors on word similarity datasets, and summarize existing solutions. Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.