# On the Information Content of Predictions in Word Analogy Tests

arXiv: https://arxiv.org/abs/2210.09972 · HTML: https://mlsanity.com/paper/2210.09972/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 27 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 7 of 7)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [Issues in evaluating semantic spaces using word analogies](https://mlsanity.com/paper/1606.07736/) (2016)
- [Problems With Evaluation of Word Embeddings Using Word Similarity Tasks](https://mlsanity.com/paper/1605.02276/) (2016)
- [Analogies minus analogy test: measuring regularities in word embeddings](https://mlsanity.com/paper/2010.03446/) (2020)
- [An Analysis of Hierarchical Text Classification Using Word Embeddings](https://mlsanity.com/paper/1809.01771/) (2018)

## Abstract (arXiv)

An approach is proposed to quantify, in bits of information, the actual relevance of analogies in analogy tests. The main component of this approach is a softaccuracy estimator that also yields entropy estimates with compensated biases. Experimental results obtained with pre-trained GloVe 300-D vectors and two public analogy test sets show that proximity hints are much more relevant than analogies in analogy tests, from an information content perspective. Accordingly, a simple word embedding model is used to predict that analogies carry about one bit of information, which is experimentally corroborated.