mlsanityData to 7 Oct 2026

Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias

arXiv 2206.09860 ↗First posted 20 Jun 2022cs.CLcited by 33

Not on the map. Why?

Abstract (arXiv)

The size of pretrained models is increasing, and so is their performance on a variety of NLP tasks. However, as their memorization capacity grows, they might pick up more social biases. In this work, we examine the connection between model size and its gender bias (specifically, occupational gender bias). We measure bias in three masked language model families (RoBERTa, DeBERTa, and T5) in two setups: directly using prompt based method, and using a downstream task (Winogender). We find on the one hand that larger models receive higher bias scores on the former task, but when evaluated on the latter, they make fewer gender errors. To examine these potentially conflicting results, we carefully investigate the behavior of the different models on Winogender. We find that while larger models outperform smaller ones, the probability that their mistakes are caused by gender bias is higher. Moreover, we find that the proportion of stereotypical errors compared to anti-stereotypical ones grows with the model size. Our findings highlight the potential risks that can arise from increasing model size.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 20 Jun 2022.

Builds on

23 of 38 references.

References · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  8. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  11. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  12. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
11 more
  1. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  2. 2018Gender Bias in Coreference Resolutioncited by 438
  3. 2019On Measuring Social Biases in Sentence Encoderscited by 397
  4. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  5. 2019Measuring Bias in Contextualized Word Representationscited by 275
  6. 2019Evaluating Gender Bias in Machine Translationcited by 247
  7. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  8. 2022Data Contamination: From Memorization to Exploitationcited by 234
  9. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  10. 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
  11. 2021Evaluating Gender Bias in Natural Language Inferencecited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 10; 2024: 10; 2025: 7; 2026: 3.

33 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  3. 2022Training Trajectories of Language Models Across Scalescited by 80
  4. 2023Instructed to Bias: Instruction-Tuned Language Models Exhibit Emergent Cognitive Biascited by 51
  5. 2023A Trip Towards Fairness: Bias and De-Biasing in Large Language Modelscited by 32
  6. 2023Sociodemographic Bias in Language Models: A Survey and Forward Pathcited by 27
  7. 2024Beyond Performance: Quantifying and Mitigating Label Bias in LLMscited by 24
  8. 2024Robust Pronoun Fidelity with English LLMs: Are they Reasoning, Repeating, or Just Biased?cited by 17
  9. 2023Fairness in Language Models Beyond English: Gaps and Challengescited by 16
  10. 2024PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Modelscited by 16
  11. 2023Women Are Beautiful, Men Are Leaders: Gender Stereotypes in Machine Translation and Language Modelingcited by 12
  12. 2022JASMINE: Arabic GPT Models for Few-Shot Learningcited by 11
21 more
  1. 2024Risks, Causes, and Mitigations of Widespread Deployments of Large Language Models (LLMs): A Surveycited by 8
  2. 2024Native Design Bias: Studying the Impact of English Nativeness on Language Model Performancecited by 5
  3. 2025Assumed Identities: Quantifying Gender Bias in Machine Translation of Gender-Ambiguous Occupational Termscited by 5
  4. 2023A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Modelscited by 4
  5. 2024What the Weight?! A Unified Framework for Zero-Shot Knowledge Compositioncited by 4
  6. 2024Investigating Markers and Drivers of Gender Bias in Machine Translationscited by 4
  7. 2024RuBia: A Russian Language Bias Detection Datasetcited by 4
  8. 2025On the Mutual Influence of Gender and Occupation in LLM Representationscited by 4
  9. 2025EuroGEST: Investigating gender stereotypes in multilingual language modelscited by 4
  10. 2025Probing Association Biases in LLM Moderation Over-Sensitivitycited by 3
  11. 2025McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Modelscited by 3
  12. 2024The Trade-off between Performance, Efficiency, and Fairness in Adapter Modules for Text Classificationcited by 2
  13. 2022Gender Biases Unexpectedly Fluctuate in the Pre-training Stage of Masked Language Modelscited by 1
  14. 2023Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Modelscited by 1
  15. 2024Investigating Gender Bias in Turkish Language Modelscited by 1
  16. 2025Fairness Dynamics During Trainingcited by 1
  17. 2023Keeping Up with the Language Models: Systematic Benchmark Extension for Bias Auditingcited by 0
  18. 2025Stereotype Detection as a Catalyst for Enhanced Bias Detection: A Multi-Task Learning Approachcited by 0
  19. 2026To Compare, or Not to Compare: On Methodological Practices in Evaluating Social Biascited by 0
  20. 2026Harm Laundering in GPT Models: Evidence That Gender Discrimination Is Transformed Rather Than Reduced Across Safety-Trained Generationscited by 0
  21. 2026Benchmarking Gender Bias in Machine Translation Evaluation Metrics across Occupationscited by 0