# Gender Bias in Masked Language Models for Multiple Languages

arXiv: https://arxiv.org/abs/2205.00551 · HTML: https://mlsanity.com/paper/2205.00551/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 54 references that are in the map.
- First posted on arXiv 1 May 2022.

## Builds on (most cited first, 30 of 36)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [On Measuring and Mitigating Biased Inferences of Word Embeddings](https://mlsanity.com/paper/1908.09369/) (2019)
- [Debiasing Pre-trained Contextualised Embeddings](https://mlsanity.com/paper/2101.09523/) (2021)
- [ConvBERT: Improving BERT with Span-based Dynamic Convolution](https://mlsanity.com/paper/2008.02496/) (2020)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings](https://mlsanity.com/paper/1811.01136/) (2018)
- [Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias](https://mlsanity.com/paper/2010.14534/) (2020)
- [Attenuating Bias in Word Vectors](https://mlsanity.com/paper/1901.07656/) (2019)
- [Gender-preserving Debiasing for Pre-trained Word Embeddings](https://mlsanity.com/paper/1906.00742/) (2019)
- [Unmasking the Mask -- Evaluating Social Biases in Masked Language Models](https://mlsanity.com/paper/2104.07496/) (2021)
- [Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00699/) (2020)
- [What's in a Name? Reducing Bias in Bios without Access to Protected Attributes](https://mlsanity.com/paper/1904.05233/) (2019)

## Built on by (most cited first, 30 of 60)

- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Tasks](https://mlsanity.com/paper/2210.02938/) (2022)
- [Should We Respect LLMs? A Cross-Lingual Study on the Influence of Prompt Politeness on LLM Performance](https://mlsanity.com/paper/2402.14531/) (2024)
- [Comparing Biases and the Impact of Multilingual Training across Multiple Languages](https://mlsanity.com/paper/2305.11242/) (2023)
- [Gender Bias in Large Language Models across Multiple Languages](https://mlsanity.com/paper/2403.00277/) (2024)
- [MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs](https://mlsanity.com/paper/2406.07243/) (2024)
- [On Evaluating and Mitigating Gender Biases in Multilingual Settings](https://mlsanity.com/paper/2307.01503/) (2023)
- [Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implications](https://mlsanity.com/paper/2307.09162/) (2023)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [An Efficient Multilingual Language Model Compression through Vocabulary Trimming](https://mlsanity.com/paper/2305.15020/) (2023)
- [Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology](https://mlsanity.com/paper/2405.06346/) (2024)
- [Fairness in Language Models Beyond English: Gaps and Challenges](https://mlsanity.com/paper/2302.12578/) (2023)
- [Evaluating Gender Bias of Pre-trained Language Models in Natural Language Inference by Considering All Labels](https://mlsanity.com/paper/2309.09697/) (2023)
- [MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2212.10455/) (2022)
- [Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflicts](https://mlsanity.com/paper/2410.11084/) (2024)
- [Quite Good, but Not Enough: Nationality Bias in Large Language Models -- A Case Study of ChatGPT](https://mlsanity.com/paper/2405.06996/) (2024)
- [Do Multilingual Large Language Models Mitigate Stereotype Bias?](https://mlsanity.com/paper/2407.05740/) (2024)
- [In-Contextual Gender Bias Suppression for Large Language Models](https://mlsanity.com/paper/2309.07251/) (2023)
- [Multilingual large language models leak human stereotypes across language boundaries](https://mlsanity.com/paper/2312.07141/) (2023)
- [The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuning](https://mlsanity.com/paper/2405.12744/) (2024)
- [DAIQ: Auditing Demographic Attribute Inference from Question in LLMs](https://mlsanity.com/paper/2508.15830/) (2025)
- [Comparing Intrinsic Gender Bias Evaluation Measures without using Human Annotated Examples](https://mlsanity.com/paper/2301.12074/) (2023)
- [In-Depth Look at Word Filling Societal Bias Measures](https://mlsanity.com/paper/2302.12640/) (2023)
- [The Impact of Debiasing on the Performance of Language Models in Downstream Tasks is Underestimated](https://mlsanity.com/paper/2309.09092/) (2023)
- [Controlled Generation with Prompt Insertion for Natural Language Explanations in Grammatical Error Correction](https://mlsanity.com/paper/2309.11439/) (2023)
- [Gender Bias in Meta-Embeddings](https://mlsanity.com/paper/2205.09867/) (2022)
- [Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings](https://mlsanity.com/paper/2403.14001/) (2024)
- [Solving Cosine Similarity Underestimation between High Frequency Words by L2 Norm Discounting](https://mlsanity.com/paper/2305.10610/) (2023)
- [Solving NLP Problems through Human-System Collaboration: A Discussion-based Approach](https://mlsanity.com/paper/2305.11789/) (2023)
- [The Gaps between Pre-train and Downstream Settings in Bias Evaluation and Debiasing](https://mlsanity.com/paper/2401.08511/) (2024)

## Abstract (arXiv)

Masked Language Models (MLMs) pre-trained by predicting masked tokens on large corpora have been used successfully in natural language processing tasks for a variety of languages. Unfortunately, it was reported that MLMs also learn discriminative biases regarding attributes such as gender and race. Because most studies have focused on MLMs in English, the bias of MLMs in other languages has rarely been investigated. Manual annotation of evaluation data for languages other than English has been challenging due to the cost and difficulty in recruiting annotators. Moreover, the existing bias evaluation methods require the stereotypical sentence pairs consisting of the same context with attribute words (e.g. He/She is a nurse). We propose Multilingual Bias Evaluation (MBE) score, to evaluate bias in various languages using only English attribute word lists and parallel corpora between the target language and English without requiring manually annotated data. We evaluated MLMs in eight languages using the MBE and confirmed that gender-related biases are encoded in MLMs for all those languages. We manually created datasets for gender bias in Japanese and Russian to evaluate the validity of the MBE. The results show that the bias scores reported by the MBE significantly correlates with that computed from the above manually created datasets and the existing English datasets for gender bias.