mlsanityData to 7 Oct 2026

Problems With Evaluation of Word Embeddings Using Word Similarity Tasks

arXiv 1605.02276 ↗First posted 8 May 2016cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Lacking standardized extrinsic evaluation methods for vector representations of words, the NLP community has relied heavily on word similarity tasks as a proxy for intrinsic evaluation of word vectors. Word similarity evaluation, which correlates the distance between vectors and human judgments of semantic similarity is attractive, because it is computationally inexpensive and fast. In this paper we present several problems associated with the evaluation of word vectors on word similarity datasets, and summarize existing solutions. Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 May 2016.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 7; 2022: 6; 2023: 4; 2024: 4; 2025: 4.

25 papers.

Sort:

  1. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  2. 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
  3. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  4. 2021Obtaining Better Static Word Embeddings Using Contextual Embedding Modelscited by 18
  5. 2022Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Costcited by 9
  6. 2021Multi-sense embeddings through a word sense disambiguation processcited by 7
  7. 2023Enhancing Interpretability using Human Similarity Judgements to Prune Word Embeddingscited by 6
  8. 2021Theoretical foundations and limits of word embeddings: what types of meaning can they capture?cited by 5
  9. 2022Learning Meta Word Embeddings by Unsupervised Weighted Concatenation of Source Embeddingscited by 5
  10. 2022No Word Embedding Model Is Perfect: Evaluating the Representation Accuracy for Social Bias in the Mediacited by 5
  11. 2022It's AI Match: A Two-Step Approach for Schema Matching Using Embeddingscited by 4
  12. 2023Multi-Relational Hyperbolic Word Embeddings from Natural Language Definitionscited by 4
13 more
  1. 2025Fine-Tuned LLMs are "Time Capsules" for Tracking Societal Bias Through Bookscited by 4
  2. 2023Is Probing All You Need? Indicator Tasks as an Alternative to Probing Embedding Spacescited by 2
  3. 2024A Comprehensive Analysis of Static Word Embeddings for Turkishcited by 2
  4. 2021SuperSim: a test set for word similarity and relatedness in Swedishcited by 1
  5. 2025Semantic Certainty Assessment in Vector Retrieval Systems: A Novel Framework for Embedding Quality Evaluationcited by 1
  6. 2025Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attentioncited by 1
  7. 2021Understanding and Enhancing the Use of Context for Machine Translationcited by 0
  8. 2021An empirical analysis of phrase-based and neural machine translationcited by 0
  9. 2021Term Interrelations and Trends in Software Engineeringcited by 0
  10. 2022On the Information Content of Predictions in Word Analogy Testscited by 0
  11. 2024Word Embedding Dimension Reduction via Weakly-Supervised Feature Selectioncited by 0
  12. 2024Interpretable Syntactic Representations Enable Hierarchical Word Vectorscited by 0
  13. 2025Set-Theoretic Compositionality of Sentence Embeddingscited by 0