mlsanityData to 7 Oct 2026

Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models

arXiv 2101.09688 ↗First posted 24 Jan 2021cs.CL cs.AI cs.LGcited by 57

Not on the map. Why?

Abstract (arXiv)

This paper proposes two intuitive metrics, skew and stereotype, that quantify and analyse the gender bias present in contextual language models when tackling the WinoBias pronoun resolution task. We find evidence that gender stereotype correlates approximately negatively with gender skew in out-of-the-box models, suggesting that there is a trade-off between these two forms of bias. We investigate two methods to mitigate bias. The first approach is an online method which is effective at removing skew at the expense of stereotype. The second, inspired by previous work on ELMo, involves the fine-tuning of BERT using an augmented gender-balanced dataset. We show that this reduces both skew and stereotype relative to its unaugmented fine-tuned counterpart. However, we find that existing gender bias benchmarks do not fully probe professional bias as pronoun resolution may be obfuscated by cross-correlations from other manifestations of gender prejudice. Our code is available online, at https://github.com/12kleingordon34/NLP_masters_project.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 25 of its 34 references that are in the map. First posted on arXiv 24 Jan 2021.

Builds on

25 of 34 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  6. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  7. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  10. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  11. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  12. 2018Gender Bias in Coreference Resolutioncited by 438
13 more
  1. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  2. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  3. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  4. 2019Measuring Bias in Contextualized Word Representationscited by 275
  5. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  6. 2017End-to-end Neural Coreference Resolutioncited by 240
  7. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  8. 2018Gender Bias in Neural Natural Language Processingcited by 210
  9. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  10. 2018Learning Gender-Neutral Word Embeddingscited by 169
  11. 2018Higher-order Coreference Resolution with Coarse-to-fine Inferencecited by 150
  12. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  13. 2016Women Through the Glass Ceiling: Gender Asymmetries in Wikipediacited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 11; 2023: 11; 2024: 16; 2025: 9; 2026: 1.

57 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  3. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  4. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  5. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  6. 2021DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networkscited by 109
  7. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  8. 2024Large Language Models are Geographically Biasedcited by 91
  9. 2021A Survey on Gender Bias in Natural Language Processingcited by 89
  10. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  11. 2023Synthcity: facilitating innovative use cases of synthetic data in different data modalitiescited by 76
  12. 2022Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Listscited by 54
38 more
  1. 2024Bias in Language Models: Beyond Trick Tests and Toward RUTEd Evaluationcited by 41
  2. 2021Your fairness may vary: Pretrained language model fairness in toxic text classificationcited by 38
  3. 2021Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgettingcited by 34
  4. 2022Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Biascited by 33
  5. 2024Interpreting Arithmetic Mechanism in Large Language Models through Comparative Neuron Analysiscited by 31
  6. 2022Choose Your Lenses: Flaws in Gender Bias Evaluationcited by 29
  7. 2023Beyond Privacy: Navigating the Opportunities and Challenges of Synthetic Datacited by 29
  8. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  9. 2021Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue modelscited by 23
  10. 2024Fairness and Bias in Multimodal AI: A Surveycited by 19
  11. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  12. 2021Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you?cited by 16
  13. 2022Don't Forget About Pronouns: Removing Gender Bias in Language Models Without Losing Factual Gender Informationcited by 16
  14. 2024Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflictscited by 15
  15. 2023Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modelingcited by 12
  16. 2022Conditional Supervised Contrastive Learning for Fair Text Classificationcited by 11
  17. 2023FairSISA: Ensemble Post-Processing to Improve Fairness of Unlearning in LLMscited by 10
  18. 2022What Do Compressed Multilingual Machine Translation Models Forget?cited by 9
  19. 2022Towards an Enhanced Understanding of Bias in Pre-trained Neural Language Models: A Survey with Special Emphasis on Affective Biascited by 8
  20. 2021Evaluating Metrics for Bias in Word Embeddingscited by 7
  21. 2025Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Modelscited by 6
  22. 2025Does the Prompt-based Large Language Model Recognize Students' Demographics and Introduce Bias in Essay Scoring?cited by 5
  23. 2025The World According to LLMs: How Geographic Origin Influences LLMs' Entity Deduction Capabilitiescited by 5
  24. 2022Conceptor-Aided Debiasing of Large Language Modelscited by 4
  25. 2024Investigating Markers and Drivers of Gender Bias in Machine Translationscited by 4
  26. 2021LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLIcited by 3
  27. 2024From Prejudice to Parity: A New Approach to Debiasing Large Language Model Word Embeddingscited by 3
  28. 2025Ontologies in Design: How Imagining a Tree Reveals Possibilites and Assumptions in Large Language Modelscited by 3
  29. 2025Mitigating Confounding in Speech-Based Dementia Detection through Weight Maskingcited by 3
  30. 2023Gender Bias in Transformer Models: A comprehensive surveycited by 2
  31. 2024How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Modelscited by 2
  32. 2024GeniL: A Multilingual Dataset on Generalizing Languagecited by 2
  33. 2024The Silent Curriculum: How Does LLM Monoculture Shape Educational Content and Its Accessibility?cited by 2
  34. 2024REFINE-LM: Mitigating Language Model Stereotypes via Reinforcement Learningcited by 2
  35. 2023Counteracts: Testing Stereotypical Representation in Pre-trained Language Modelscited by 1
  36. 2023A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classificationcited by 1
  37. 2023Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Modelscited by 1
  38. 2025MASS: Overcoming Language Bias in Image-Text Matchingcited by 1

50 most cited of 57 All 57 (JSON)