# On Evaluating and Mitigating Gender Biases in Multilingual Settings

arXiv: https://arxiv.org/abs/2307.01503 · HTML: https://mlsanity.com/paper/2307.01503/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 4 Jul 2023.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Revisiting Pre-Trained Models for Chinese Natural Language Processing](https://mlsanity.com/paper/2004.13922/) (2020)
- [Measuring and Reducing Gendered Correlations in Pre-trained Models](https://mlsanity.com/paper/2010.06032/) (2020)
- [Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models](https://mlsanity.com/paper/2101.00288/) (2021)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [Sustainable Modular Debiasing of Language Models](https://mlsanity.com/paper/2109.03646/) (2021)
- [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://mlsanity.com/paper/2110.08527/) (2021)
- [Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models](https://mlsanity.com/paper/2204.08110/) (2022)
- [Mitigating Language-Dependent Ethnic Bias in BERT](https://mlsanity.com/paper/2109.05704/) (2021)
- [Gender Bias in Masked Language Models for Multiple Languages](https://mlsanity.com/paper/2205.00551/) (2022)
- [Gender Bias in Multilingual Embeddings and Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00699/) (2020)
- [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022)
- [Text Generation with Text-Editing Models](https://mlsanity.com/paper/2206.07043/) (2022)

## Built on by (most cited first, 30 of 31)

- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm](https://mlsanity.com/paper/2406.18682/) (2024)
- [RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs](https://mlsanity.com/paper/2407.02552/) (2024)
- [How Does Quantization Affect Multilingual LLMs?](https://mlsanity.com/paper/2407.03211/) (2024)
- [MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs](https://mlsanity.com/paper/2406.07243/) (2024)
- [SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes](https://mlsanity.com/paper/2403.05696/) (2024)
- [Indian-BhED: A Dataset for Measuring India-Centric Biases in Large Language Models](https://mlsanity.com/paper/2309.08573/) (2023)
- [BiasEdit: Debiasing Stereotyped Language Models via Model Editing](https://mlsanity.com/paper/2503.08588/) (2025)
- [From 'Showgirls' to 'Performers': Fine-tuning with Gender-inclusive Language for Bias Reduction in LLMs](https://mlsanity.com/paper/2407.04434/) (2024)
- [Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Bias](https://mlsanity.com/paper/2407.03536/) (2024)
- [Do Multilingual Large Language Models Mitigate Stereotype Bias?](https://mlsanity.com/paper/2407.05740/) (2024)
- [Executable Counterfactuals: Improving LLMs' Causal Reasoning Through Code](https://mlsanity.com/paper/2510.01539/) (2025)
- [GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models](https://mlsanity.com/paper/2408.12494/) (2024)
- [The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context](https://mlsanity.com/paper/2504.02708/) (2025)
- [Bhaasha, Bhasa, Zaban: A Survey for Low-Resourced Languages in South Asia -- Current Stage and Challenges](https://mlsanity.com/paper/2509.11570/) (2025)
- [Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation](https://mlsanity.com/paper/2412.14050/) (2024)
- [The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models](https://mlsanity.com/paper/2602.11174/) (2026)
- [Different Bias Under Different Criteria: Assessing Bias in LLMs with a Fact-Based Approach](https://mlsanity.com/paper/2411.17338/) (2024)
- [CONGRAD:Conflicting Gradient Filtering for Multilingual Preference Alignment](https://mlsanity.com/paper/2503.23777/) (2025)
- [Exploring Gender Bias in Large Language Models: An In-depth Dive into the German Language](https://mlsanity.com/paper/2507.16557/) (2025)
- [Beyond Generative Artificial Intelligence: Roadmap for Natural Language Generation](https://mlsanity.com/paper/2407.10554/) (2024)
- [Social Bias in Multilingual Language Models: A Survey](https://mlsanity.com/paper/2508.20201/) (2025)
- [IndiCASA: A Dataset and Bias Evaluation Framework in LLMs Using Contrastive Embedding Similarity in the Indian Context](https://mlsanity.com/paper/2510.02742/) (2025)
- [Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages](https://mlsanity.com/paper/2510.05291/) (2025)
- [Current State-of-the-Art of Bias Detection and Mitigation in Machine Translation for African and European Languages: a Review](https://mlsanity.com/paper/2410.21126/) (2024)
- [Iterative Multilingual Spectral Attribute Erasure](https://mlsanity.com/paper/2506.11244/) (2025)
- [Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration](https://mlsanity.com/paper/2607.07895/) (2026)
- [Assessing Agentic Large Language Models in Multilingual National Bias](https://mlsanity.com/paper/2502.17945/) (2025)
- [Teaching and Critiquing Conceptualization and Operationalization in NLP](https://mlsanity.com/paper/2512.18505/) (2025)

## Abstract (arXiv)

While understanding and removing gender biases in language models has been a long-standing problem in Natural Language Processing, prior research work has primarily been limited to English. In this work, we investigate some of the challenges with evaluating and mitigating biases in multilingual settings which stem from a lack of existing benchmarks and resources for bias evaluation beyond English especially for non-western context. In this paper, we first create a benchmark for evaluating gender biases in pre-trained masked language models by extending DisCo to different Indian languages using human annotations. We extend various debiasing methods to work beyond English and evaluate their effectiveness for SOTA massively multilingual models on our proposed metric. Overall, our work highlights the challenges that arise while studying social biases in multilingual settings and provides resources as well as mitigation techniques to take a step toward scaling to more languages.