# The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks

arXiv: https://arxiv.org/abs/2210.10040 · HTML: https://mlsanity.com/paper/2210.10040/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 54 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 30 of 45)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [A Decomposable Attention Model for Natural Language Inference](https://mlsanity.com/paper/1606.01933/) (2016)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Higher-order Coreference Resolution with Coarse-to-fine Inference](https://mlsanity.com/paper/1804.05392/) (2018)
- [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://mlsanity.com/paper/2012.15859/) (2020)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [UnQovering Stereotyping Biases via Underspecified Questions](https://mlsanity.com/paper/2010.02428/) (2020)
- [Correcting Length Bias in Neural Machine Translation](https://mlsanity.com/paper/1808.10006/) (2018)

## Built on by (most cited first, 30 of 31)

- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [The Bias Amplification Paradox in Text-to-Image Generation](https://mlsanity.com/paper/2308.00755/) (2023)
- [MISGENDERED: Limits of Large Language Models in Understanding Pronouns](https://mlsanity.com/paper/2306.03950/) (2023)
- [KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledge](https://mlsanity.com/paper/2402.13605/) (2024)
- [Sociodemographic Bias in Language Models: A Survey and Forward Path](https://mlsanity.com/paper/2306.08158/) (2023)
- [ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation](https://mlsanity.com/paper/2401.06310/) (2024)
- [It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performance](https://mlsanity.com/paper/2305.09022/) (2023)
- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Models](https://mlsanity.com/paper/2312.07492/) (2023)
- [Perceptions of Linguistic Uncertainty by Language Models and Humans](https://mlsanity.com/paper/2407.15814/) (2024)
- [CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Bias](https://mlsanity.com/paper/2308.12539/) (2023)
- [Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modeling](https://mlsanity.com/paper/2311.18711/) (2023)
- [Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societies](https://mlsanity.com/paper/2312.01509/) (2023)
- [Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitations](https://mlsanity.com/paper/2403.06009/) (2024)
- [Are Models Biased on Text without Gender-related Language?](https://mlsanity.com/paper/2405.00588/) (2024)
- ["Cold, Calculated, and Condescending": How AI Identifies and Explains Ableism Compared to Disabled People](https://mlsanity.com/paper/2410.03448/) (2024)
- [WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Case](https://mlsanity.com/paper/2409.05653/) (2024)
- [BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Models](https://mlsanity.com/paper/2302.07371/) (2023)
- [Uncovering Political Bias in Large Language Models using Parliamentary Voting Records](https://mlsanity.com/paper/2601.08785/) (2026)
- [Agree to Disagree? A Meta-Evaluation of LLM Misgendering](https://mlsanity.com/paper/2504.17075/) (2025)
- [Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspective](https://mlsanity.com/paper/2303.09092/) (2023)
- [How far can bias go? Tracing bias from pretraining data to alignment](https://mlsanity.com/paper/2411.19240/) (2024)
- [Characterizing Mamba's Selective Memory using Auto-Encoders](https://mlsanity.com/paper/2512.15653/) (2025)
- [Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Bias](https://mlsanity.com/paper/2601.18486/) (2026)
- [Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasets](https://mlsanity.com/paper/2501.01168/) (2025)
- [A Scalable Entity-Based Framework for Auditing Bias in Large Language Models](https://mlsanity.com/paper/2601.12374/) (2026)
- [Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditing](https://mlsanity.com/paper/2305.12620/) (2023)
- [When in Doubt, Cascade: Towards Building Efficient and Capable Guardrails](https://mlsanity.com/paper/2407.06323/) (2024)
- [EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering](https://mlsanity.com/paper/2507.11216/) (2025)

## Abstract (arXiv)

How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given language model? In this work, we study this question by contrasting social biases with non-social biases stemming from choices made during dataset construction that might not even be discernible to the human eye. To do so, we empirically simulate various alternative constructions for a given benchmark based on innocuous modifications (such as paraphrasing or random-sampling) that maintain the essence of their social bias. On two well-known social bias benchmarks (Winogender and BiasNLI) we observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models. We hope these troubling observations motivate more robust measures of social biases.