# How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks

arXiv: https://arxiv.org/abs/1702.02170 · HTML: https://mlsanity.com/paper/1702.02170/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2017.

## Built on by (most cited first, 12 of 12)

- [DirectProbe: Studying Representations without Classifiers](https://mlsanity.com/paper/2104.05904/) (2021)
- [A comparative evaluation and analysis of three generations of Distributional Semantic Models](https://mlsanity.com/paper/2105.09825/) (2021)
- [Marked Attribute Bias in Natural Language Inference](https://mlsanity.com/paper/2109.14039/) (2021)
- [Axis Tour: Word Tour Determines the Order of Axes in ICA-transformed Embeddings](https://mlsanity.com/paper/2401.06112/) (2024)
- [Revisiting Cosine Similarity via Normalized ICA-transformed Embeddings](https://mlsanity.com/paper/2406.10984/) (2024)
- [IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level Grounding of Images](https://mlsanity.com/paper/2305.10438/) (2023)
- [A New Pair of GloVes](https://mlsanity.com/paper/2507.18103/) (2025)
- [Word-Graph2vec: An efficient word embedding approach on word co-occurrence graph using random walk technique](https://mlsanity.com/paper/2301.04312/) (2023)
- [Social-Group-Agnostic Word Embedding Debiasing via the Stereotype Content Model](https://mlsanity.com/paper/2210.05831/) (2022)
- [Detecting and Mitigating Indirect Stereotypes in Word Embeddings](https://mlsanity.com/paper/2305.14574/) (2023)
- [RESTORE: Graph Embedding Assessment Through Reconstruction](https://mlsanity.com/paper/2308.14659/) (2023)
- [RiverText: A Python Library for Training and Evaluating Incremental Word Embeddings from Text Data Streams](https://mlsanity.com/paper/2506.23192/) (2025)

## Abstract (arXiv)

Maybe the single most important goal of representation learning is making subsequent learning faster. Surprisingly, this fact is not well reflected in the way embeddings are evaluated. In addition, recent practice in word embeddings points towards importance of learning specialized representations. We argue that focus of word representation evaluation should reflect those trends and shift towards evaluating what useful information is easily accessible. Specifically, we propose that evaluation should focus on data efficiency and simple supervised tasks, where the amount of available data is varied and scores of a supervised model are reported for each subset (as commonly done in transfer learning). In order to illustrate significance of such analysis, a comprehensive evaluation of selected word embeddings is presented. Proposed approach yields a more complete picture and brings new insight into performance characteristics, for instance information about word similarity or analogy tends to be non--linearly encoded in the embedding space, which questions the cosine-based, unsupervised, evaluation methods. All results and analysis scripts are available online.