# Investigating the Frequency Distortion of Word Embeddings and Its Impact on Bias Metrics

arXiv: https://arxiv.org/abs/2211.08203 · HTML: https://mlsanity.com/paper/2211.08203/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 15 Nov 2022.

## Builds on (most cited first, 11 of 11)

- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypes](https://mlsanity.com/paper/1711.08412/) (2017)
- [All-but-the-Top: Simple and Effective Postprocessing for Word Representations](https://mlsanity.com/paper/1702.01417/) (2017)
- [The Geometry of Culture: Analyzing Meaning through Word Embeddings](https://mlsanity.com/paper/1803.09288/) (2018)
- [FRAGE: Frequency-Agnostic Word Representation](https://mlsanity.com/paper/1809.06858/) (2018)
- [Comparative study of LSA vs Word2vec embeddings in small corpora: a case study in dreams database](https://mlsanity.com/paper/1610.01520/) (2016)
- [A Survey On Neural Word Embeddings](https://mlsanity.com/paper/2110.01804/) (2021)

## Built on by (most cited first, 2 of 2)

- [Understanding Token Probability Encoding in Output Embeddings](https://mlsanity.com/paper/2406.01468/) (2024)
- [Geometric Structure Predicts Per-Example Neighborhood Loss Under PCA Truncation](https://mlsanity.com/paper/2610.17742/) (2026)

## Abstract (arXiv)

Recent research has shown that static word embeddings can encode word frequency information. However, little has been studied about this phenomenon and its effects on downstream tasks. In the present work, we systematically study the association between frequency and semantic similarity in several static word embeddings. We find that Skip-gram, GloVe and FastText embeddings tend to produce higher semantic similarity between high-frequency words than between other frequency combinations. We show that the association between frequency and similarity also appears when words are randomly shuffled. This proves that the patterns found are not due to real semantic associations present in the texts, but are an artifact produced by the word embeddings. Finally, we provide an example of how word frequency can strongly impact the measurement of gender bias with embedding-based metrics. In particular, we carry out a controlled experiment that shows that biases can even change sign or reverse their order by manipulating word frequencies.