# Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairs

arXiv: https://arxiv.org/abs/2010.02847 · HTML: https://mlsanity.com/paper/2010.02847/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 6 of 6)

- [Assessing the Reliability of Word Embedding Gender Bias Measures](https://mlsanity.com/paper/2109.04732/) (2021)
- [WordBias: An Interactive Visual Tool for Discovering Intersectional Biases Encoded in Word Embeddings](https://mlsanity.com/paper/2103.03598/) (2021)
- [Trustworthy Social Bias Measurement](https://mlsanity.com/paper/2212.11672/) (2022)
- [Undesirable Biases in NLP: Addressing Challenges of Measurement](https://mlsanity.com/paper/2211.13709/) (2022)
- [Sexism in the Judiciary](https://mlsanity.com/paper/2106.15103/) (2021)
- [Towards Fair and Explainable AI using a Human-Centered AI Approach](https://mlsanity.com/paper/2306.07427/) (2023)

## Abstract (arXiv)

It has been shown that word embeddings can exhibit gender bias, and various methods have been proposed to quantify this. However, the extent to which the methods are capturing social stereotypes inherited from the data has been debated. Bias is a complex concept and there exist multiple ways to define it. Previous work has leveraged gender word pairs to measure bias and extract biased analogies. We show that the reliance on these gendered pairs has strong limitations: bias measures based off of them are not robust and cannot identify common types of real-world bias, whilst analogies utilising them are unsuitable indicators of bias. In particular, the well-known analogy "man is to computer-programmer as woman is to homemaker" is due to word similarity rather than societal bias. This has important implications for work on measuring bias in embeddings and related work debiasing embeddings.