# Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias

arXiv: https://arxiv.org/abs/2010.14534 · HTML: https://mlsanity.com/paper/2010.14534/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2020.

## Built on by (most cited first, 30 of 82)

- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Sustainable Modular Debiasing of Language Models](https://mlsanity.com/paper/2109.03646/) (2021)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Gender Bias in Masked Language Models for Multiple Languages](https://mlsanity.com/paper/2205.00551/) (2022)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists](https://mlsanity.com/paper/2203.09192/) (2022)
- [Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks](https://mlsanity.com/paper/2210.02938/) (2022)
- [Benefits and Harms of Large Language Models in Digital Mental Health](https://mlsanity.com/paper/2311.14693/) (2023)
- [MABEL: Attenuating Gender Bias using Textual Entailment Data](https://mlsanity.com/paper/2210.14975/) (2022)
- [Choose Your Lenses: Flaws in Gender Bias Evaluation](https://mlsanity.com/paper/2210.11471/) (2022)
- [On the Independence of Association Bias and Empirical Fairness in Language Models](https://mlsanity.com/paper/2304.10153/) (2023)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [Understanding and Countering Stereotypes: A Computational Approach to the Stereotype Content Model](https://mlsanity.com/paper/2106.02596/) (2021)
- [Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task](https://mlsanity.com/paper/2306.15298/) (2023)
- [Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models](https://mlsanity.com/paper/2112.07447/) (2021)
- [Fairness through Difference Awareness: Measuring Desired Group Discrimination in LLMs](https://mlsanity.com/paper/2502.01926/) (2025)
- [Naturalistic Causal Probing for Morpho-Syntax](https://mlsanity.com/paper/2205.07043/) (2022)
- [Large Language Model Bias Mitigation from the Perspective of Knowledge Editing](https://mlsanity.com/paper/2405.09341/) (2024)
- [Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You](https://mlsanity.com/paper/2401.16092/) (2024)
- [CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias](https://mlsanity.com/paper/2308.12539/) (2023)
- [Identifying and Mitigating Social Bias Knowledge in Language Models](https://mlsanity.com/paper/2408.11843/) (2024)
- [Detecting Unintended Social Bias in Toxic Language Datasets](https://mlsanity.com/paper/2210.11762/) (2022)
- [Multilingual large language models leak human stereotypes across language boundaries](https://mlsanity.com/paper/2312.07141/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [Analyzing Gender Representation in Multilingual Models](https://mlsanity.com/paper/2204.09168/) (2022)
- [Measuring Gender Bias in West Slavic Language Models](https://mlsanity.com/paper/2304.05783/) (2023)
- [Potential and Challenges of Model Editing for Social Debiasing](https://mlsanity.com/paper/2402.13462/) (2024)

## Abstract (arXiv)

Contextualized word embeddings have been replacing standard embeddings as the representational knowledge source of choice in NLP systems. Since a variety of biases have previously been found in standard word embeddings, it is crucial to assess biases encoded in their replacements as well. Focusing on BERT (Devlin et al., 2018), we measure gender bias by studying associations between gender-denoting target words and names of professions in English and German, comparing the findings with real-world workforce statistics. We mitigate bias by fine-tuning BERT on the GAP corpus (Webster et al., 2018), after applying Counterfactual Data Substitution (CDS) (Maudslay et al., 2019). We show that our method of measuring bias is appropriate for languages such as English, but not for languages with a rich morphology and gender-marking, such as German. Our results highlight the importance of investigating bias and mitigation techniques cross-linguistically, especially in view of the current emphasis on large-scale, multilingual language models.