mlsanityData to 7 Oct 2026

The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks

arXiv 2210.10040 ↗First posted 18 Oct 2022cs.CL cs.CY cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given language model? In this work, we study this question by contrasting social biases with non-social biases stemming from choices made during dataset construction that might not even be discernible to the human eye. To do so, we empirically simulate various alternative constructions for a given benchmark based on innocuous modifications (such as paraphrasing or random-sampling) that maintain the essence of their social bias. On two well-known social bias benchmarks (Winogender and BiasNLI) we observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models. We hope these troubling observations motivate more robust measures of social biases.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Builds on 45 of its 54 references that are in the map. First posted on arXiv 18 Oct 2022.

Builds on

45 of 54 references.

References · 45

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  11. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  12. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
33 more
  1. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  2. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  3. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  4. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  5. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  6. 2018Gender Bias in Coreference Resolutioncited by 438
  7. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  8. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  9. 2016A Decomposable Attention Model for Natural Language Inferencecited by 214
  10. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  11. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  12. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  13. 2018Higher-order Coreference Resolution with Coarse-to-fine Inferencecited by 150
  14. 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
  15. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  16. 2021Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologiescited by 128
  17. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  18. 2018Correcting Length Bias in Neural Machine Translationcited by 119
  19. 2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
  20. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 83
  21. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  22. 2022On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representationscited by 75
  23. 2021Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translationcited by 62
  24. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  25. 2021Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Modelscited by 52
  26. 2021Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions?cited by 46
  27. 2020OSCaR: Orthogonal Subspace Correction and Rectification of Biases in Word Embeddingscited by 40
  28. 2021Your fairness may vary: Pretrained language model fairness in toxic text classificationcited by 38
  29. 2022Quantifying Social Biases Using Templates is Unreliablecited by 38
  30. 2021On Generalization in Coreference Resolutioncited by 24
  31. 2021Few-shot Instruction Prompts for Pretrained Language Models to Detect Social Biasescited by 21
  32. 2020Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and modelscited by 11
  33. 2021Double Perturbation: On the Robustness of Robustness and Counterfactual Bias Evaluationcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 12; 2024: 10; 2025: 4; 2026: 4.

31 papers.

Sort:

By area:

  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  3. 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 69
  4. 2023MISGENDERED: Limits of Large Language Models in Understanding Pronounscited by 43
  5. 2024KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledgecited by 33
  6. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  7. 2024ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generationcited by 27
  8. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 25
  9. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  10. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
  11. 2023SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Modelscited by 14
  12. 2023CALM : A Multi-task Benchmark for Comprehensive Assessment of Language Model Biascited by 13
19 more
  1. 2023Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modelingcited by 12
  2. 2023Tackling Bias in Pre-trained Language Models: Current Trends and Under-represented Societiescited by 11
  3. 2024Detectors for Safe and Reliable LLMs: Implementations, Uses, and Limitationscited by 10
  4. 2024Are Models Biased on Text without Gender-related Language?cited by 10
  5. 2024WinoPron: Revisiting English Winogender Schemas for Consistency, Coverage, and Grammatical Casecited by 9
  6. 2024"Cold, Calculated, and Condescending": How AI Identifies and Explains Ableism Compared to Disabled Peoplecited by 9
  7. 2023BiasTestGPT: Using ChatGPT for Social Bias Testing of Language Modelscited by 8
  8. 2026Uncovering Political Bias in Large Language Models using Parliamentary Voting Recordscited by 7
  9. 2023Challenges to Evaluating the Generalization of Coreference Resolution Models: A Measurement Modeling Perspectivecited by 6
  10. 2025Agree to Disagree? A Meta-Evaluation of LLM Misgenderingcited by 6
  11. 2024How far can bias go? Tracing bias from pretraining data to alignmentcited by 4
  12. 2025Blind Men and the Elephant: Diverse Perspectives on Gender Stereotypes in Benchmark Datasetscited by 3
  13. 2025Characterizing Mamba's Selective Memory using Auto-Encoderscited by 3
  14. 2026Different Demographic Cues Yield Inconsistent Conclusions About LLM Personalization and Biascited by 3
  15. 2026A Scalable Entity-Based Framework for Auditing Bias in Large Language Modelscited by 2
  16. 2024When in Doubt, Cascade: Towards Building Efficient and Capable Guardrailscited by 1
  17. 2025EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answeringcited by 1
  18. 2023Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditingcited by 0
  19. 2026Script Sensitivity: Benchmarking Language Models on Unicode, Romanized and Mixed-Script Sinhalacited by 0