# Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation

arXiv: https://arxiv.org/abs/2109.03858 · HTML: https://mlsanity.com/paper/2109.03858/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 8 Sep 2021.

## Builds on (most cited first, 24 of 24)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [AllenNLP: A Deep Semantic Natural Language Processing Platform](https://mlsanity.com/paper/1803.07640/) (2018)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle](https://mlsanity.com/paper/1901.10002/) (2019)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [BERT for Coreference Resolution: Baselines and Analysis](https://mlsanity.com/paper/1908.09091/) (2019)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Neural Machine Translation Doesn't Translate Gender Coreference Right Unless You Make It](https://mlsanity.com/paper/2010.05332/) (2020)
- [Automatically Identifying Gender Issues in Machine Translation using Perturbations](https://mlsanity.com/paper/2004.14065/) (2020)
- [Gender Coreference and Bias Evaluation at WMT 2020](https://mlsanity.com/paper/2010.06018/) (2020)
- [Syntactic Search by Example](https://mlsanity.com/paper/2006.03010/) (2020)
- [Investigating Failures of Automatic Translation in the Case of Unambiguous Gender](https://mlsanity.com/paper/2104.07838/) (2021)
- [Resolving Gendered Ambiguous Pronouns with BERT](https://mlsanity.com/paper/1906.01161/) (2019)
- [Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias](https://mlsanity.com/paper/2009.11982/) (2020)
- [The Gap on GAP: Tackling the Problem of Differing Data Distributions in Bias-Measuring Datasets](https://mlsanity.com/paper/2011.01837/) (2020)

## Built on by (most cited first, 30 of 62)

- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [Inseq: An Interpretability Toolkit for Sequence Generation Models](https://mlsanity.com/paper/2302.13942/) (2023)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution](https://mlsanity.com/paper/2306.12424/) (2023)
- [Social Bias Evaluation for Large Language Models Requires Prompt Variations](https://mlsanity.com/paper/2407.03129/) (2024)
- [The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks](https://mlsanity.com/paper/2210.10040/) (2022)
- [LingMess: Linguistically Informed Multi Expert Scorers for Coreference Resolution](https://mlsanity.com/paper/2205.12644/) (2022)
- [Multilingual Holistic Bias: Extending Descriptors and Patterns to Unveil Demographic Biases in Languages at Scale](https://mlsanity.com/paper/2305.13198/) (2023)
- [Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translation](https://mlsanity.com/paper/2203.09866/) (2022)
- [Identifying and Adapting Transformer-Components Responsible for Gender Bias in an English Language Model](https://mlsanity.com/paper/2310.12611/) (2023)
- [Fairness-Aware Structured Pruning in Transformers](https://mlsanity.com/paper/2312.15398/) (2023)
- [What about em? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns](https://mlsanity.com/paper/2305.16051/) (2023)
- [A Tale of Pronouns: Interpretability Informs Gender Bias Mitigation for Fairer Instruction-Tuned Machine Translation](https://mlsanity.com/paper/2310.12127/) (2023)
- [CEB: Compositional Evaluation Benchmark for Fairness in Large Language Models](https://mlsanity.com/paper/2407.02408/) (2024)
- [Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?](https://mlsanity.com/paper/2404.03134/) (2024)
- [Akal Badi ya Bias: An Exploratory Study of Gender Bias in Hindi Language Technology](https://mlsanity.com/paper/2405.06346/) (2024)
- [Large Language Models Still Exhibit Bias in Long Text](https://mlsanity.com/paper/2410.17519/) (2024)
- [''Fifty Shades of Bias'': Normative Ratings of Gender Bias in GPT Generated English Text](https://mlsanity.com/paper/2310.17428/) (2023)
- [What the Harm? Quantifying the Tangible Impact of Gender Bias in Machine Translation with a Human-centered Study](https://mlsanity.com/paper/2410.00545/) (2024)
- [The power of Prompts: Evaluating and Mitigating Gender Bias in MT with LLMs](https://mlsanity.com/paper/2407.18786/) (2024)
- [Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling](https://mlsanity.com/paper/2311.18711/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [Your Large Language Model is Secretly a Fairness Proponent and You Should Prompt it Like One](https://mlsanity.com/paper/2402.12150/) (2024)
- [Watching the Watchers: Exposing Gender Disparities in Machine Translation Quality Estimation](https://mlsanity.com/paper/2410.10995/) (2024)
- [How sensitive are translation systems to extra contexts? Mitigating gender bias in Neural Machine Translation models through relevant contexts](https://mlsanity.com/paper/2205.10762/) (2022)
- [Are Models Biased on Text without Gender-related Language?](https://mlsanity.com/paper/2405.00588/) (2024)
- [The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generation](https://mlsanity.com/paper/2501.07849/) (2025)

## Abstract (arXiv)

Recent works have found evidence of gender bias in models of machine translation and coreference resolution using mostly synthetic diagnostic datasets. While these quantify bias in a controlled experiment, they often do so on a small scale and consist mostly of artificial, out-of-distribution sentences. In this work, we find grammatical patterns indicating stereotypical and non-stereotypical gender-role assignments (e.g., female nurses versus male dancers) in corpora from three domains, resulting in a first large-scale gender bias dataset of 108K diverse real-world English sentences. We manually verify the quality of our corpus and use it to evaluate gender bias in various coreference resolution and machine translation models. We find that all tested models tend to over-rely on gender stereotypes when presented with natural inputs, which may be especially harmful when deployed in commercial systems. Finally, we show that our dataset lends itself to finetuning a coreference resolution model, finding it mitigates bias on a held out set. Our dataset and models are publicly available at www.github.com/SLAB-NLP/BUG. We hope they will spur future research into gender bias evaluation mitigation techniques in realistic settings.