mlsanityData to 7 Oct 2026

Assessing the Reliability of Word Embedding Gender Bias Measures

arXiv 2109.04732 ↗First posted 10 Sep 2021cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Various measures have been proposed to quantify human-like social biases in word embeddings. However, bias scores based on these measures can suffer from measurement error. One indication of measurement quality is reliability, concerning the extent to which a measure produces consistent results. In this paper, we assess three types of reliability of word embedding gender bias measures, namely test-retest reliability, inter-rater consistency and internal consistency. Specifically, we investigate the consistency of bias scores across different choices of random seeds, scoring rules and words. Furthermore, we analyse the effects of various factors on these measures' reliability scores. Our findings inform better design of word embedding gender bias measures. Moreover, we urge researchers to be more critical about the application of such measures.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 19 of its 48 references that are in the map. First posted on arXiv 10 Sep 2021.

Builds on

19 of 48 references.

References · 19

  1. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  2. 2016Pointer Sentinel Mixture Modelscited by 2,992
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  5. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
  6. 2019Measurement and Fairnesscited by 299
  7. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  8. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  9. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  10. 2018Learning Gender-Neutral Word Embeddingscited by 169
  11. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  12. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
7 more
  1. 2014Statistically Significant Detection of Linguistic Changecited by 82
  2. 2019Understanding Undesirable Word Embedding Associationscited by 59
  3. 2019Examining Gender Bias in Languages with Grammatical Gendercited by 40
  4. 2018Factors Influencing the Surprising Instability of Word Embeddingscited by 31
  5. 2020Exploring the Linear Subspace Hypothesis in Gender Bias Mitigationcited by 25
  6. 2020Robustness and Reliability of Gender Bias Assessment in Word Embeddings: The Role of Base Pairscited by 6
  7. 2018The Influence of Down-Sampling Strategies on SVD Word Embedding Stabilitycited by 4

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 2; 2024: 5; 2025: 2.

14 papers.

Sort:

By area:

  1. 2024Large Language Models are Geographically Biasedcited by 91
  2. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  3. 2022The Birth of Bias: A case study on the evolution of gender bias in an English language modelcited by 16
  4. 2023What Do Llamas Really Think? Revealing Preference Biases in Language Model Representationscited by 10
  5. 2022Trustworthy Social Bias Measurementcited by 7
  6. 2022Undesirable Biases in NLP: Addressing Challenges of Measurementcited by 4
  7. 2022On Text-based Personality Computing: Challenges and Future Directionscited by 3
  8. 2025Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systemscited by 3
  9. 2024A Note on Bias to Completecited by 1
  10. 2024PATCH! {P}sychometrics-{A}ssis{T}ed Ben{CH}marking of Large Language Models against Human Populations: A Case Study of Proficiency in 8th Grade Mathematicscited by 1
  11. 2024Reliability of Topic Modelingcited by 1
  12. 2022The SAME score: Improved cosine based bias score for word embeddingscited by 0
2 more
  1. 2024Semantic Properties of cosine based bias scores for word embeddingscited by 0
  2. 2025From Data-Driven to Purpose-Driven Artificial Intelligence: Systems Thinking for Data-Analytic Automation of Patient Carecited by 0