# Evaluating Word Embedding Models: Methods and Experimental Results

arXiv: https://arxiv.org/abs/1901.09785 · HTML: https://mlsanity.com/paper/1901.09785/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jan 2019.

## Built on by (most cited first, 30 of 37)

- [All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality](https://mlsanity.com/paper/2109.04404/) (2021)
- [Just Rank: Rethinking Evaluation with Word and Sentence Similarities](https://mlsanity.com/paper/2203.02679/) (2022)
- [Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations](https://mlsanity.com/paper/2104.10833/) (2021)
- [Using Language Model to Bootstrap Human Activity Recognition Ambient Sensors Based in Smart Homes](https://mlsanity.com/paper/2111.12158/) (2021)
- [Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2201.00768/) (2022)
- [Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey](https://mlsanity.com/paper/2402.17467/) (2024)
- [Imputing Out-of-Vocabulary Embeddings with LOVE Makes Language Models Robust with Little Cost](https://mlsanity.com/paper/2203.07860/) (2022)
- [Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training](https://mlsanity.com/paper/2104.07500/) (2021)
- [Summarising Historical Text in Modern Languages](https://mlsanity.com/paper/2101.10759/) (2021)
- [An evaluation of Google Translate for Sanskrit to English translation via sentiment and semantic analysis](https://mlsanity.com/paper/2303.07201/) (2023)
- [From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Models](https://mlsanity.com/paper/2411.05036/) (2024)
- [Language with Vision: a Study on Grounded Word and Sentence Embeddings](https://mlsanity.com/paper/2206.08823/) (2022)
- [From Topology to Retrieval: Decoding Embedding Spaces with Unified Signatures](https://mlsanity.com/paper/2511.22150/) (2025)
- [Regularizing Transformers With Deep Probabilistic Layers](https://mlsanity.com/paper/2108.10764/) (2021)
- [Getting BART to Ride the Idiomatic Train: Learning to Represent Idiomatic Expressions](https://mlsanity.com/paper/2207.03679/) (2022)
- [Word Embedding for Social Sciences: An Interdisciplinary Survey](https://mlsanity.com/paper/2207.03086/) (2022)
- [Semantic properties of English nominal pluralization: Insights from word embeddings](https://mlsanity.com/paper/2203.15424/) (2022)
- [Making sense of spoken plurals](https://mlsanity.com/paper/2207.01947/) (2022)
- [Gender bias in (non)-contextual clinical word embeddings for stereotypical medical categories](https://mlsanity.com/paper/2208.01341/) (2022)
- [Benchmarking and Analyzing In-context Learning, Fine-tuning and Supervised Learning for Biomedical Knowledge Curation: a focused study on chemical entities of biological interest](https://mlsanity.com/paper/2312.12989/) (2023)
- [A Comprehensive Analysis of Static Word Embeddings for Turkish](https://mlsanity.com/paper/2405.07778/) (2024)
- [Latent Structures of Intertextuality in French Fiction](https://mlsanity.com/paper/2410.17759/) (2024)
- [A Comprehensive Empirical Evaluation of Existing Word Embedding Approaches](https://mlsanity.com/paper/2303.07196/) (2023)
- [Manufacturing Service Capability Prediction with Graph Neural Networks](https://mlsanity.com/paper/2403.17239/) (2024)
- [Bias in Text Embedding Models](https://mlsanity.com/paper/2406.12138/) (2024)
- [Hybrid deep learning methods for phenotype prediction from clinical notes](https://mlsanity.com/paper/2108.10682/) (2021)
- [Regional Differences in Information Privacy Concerns After the Facebook-Cambridge Analytica Data Scandal](https://mlsanity.com/paper/2202.07075/) (2022)
- [An Evaluation Dataset for Legal Word Embedding: A Case Study On Chinese Codex](https://mlsanity.com/paper/2203.15173/) (2022)
- [How direct is the link between words and images?](https://mlsanity.com/paper/2206.15381/) (2022)
- [Beyond Accuracy: Measuring Representation Capacity of Embeddings to Preserve Structural and Contextual Information](https://mlsanity.com/paper/2309.11294/) (2023)

## Abstract (arXiv)

Extensive evaluation on a large number of word embedding models for language processing applications is conducted in this work. First, we introduce popular word embedding models and discuss desired properties of word models and evaluation methods (or evaluators). Then, we categorize evaluators into intrinsic and extrinsic two types. Intrinsic evaluators test the quality of a representation independent of specific natural language processing tasks while extrinsic evaluators use word embeddings as input features to a downstream task and measure changes in performance metrics specific to that task. We report experimental results of intrinsic and extrinsic evaluators on six word embedding models. It is shown that different evaluators focus on different aspects of word models, and some are more correlated with natural language processing tasks. Finally, we adopt correlation analysis to study performance consistency of extrinsic and intrinsic evalutors.