mlsanityData to 7 Oct 2026

StereoSet: Measuring stereotypical bias in pretrained language models

arXiv 2004.09456 ↗First posted 20 Apr 2020cs.CL cs.AI cs.CYcited by 911

Not on the map. Why?

Cited by 32 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or Asians are bad drivers. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. In order to assess the adverse effects of these models, it is important to quantify the bias captured in them. Existing literature on quantifying bias evaluates pretrained language models on a small set of artificially constructed bias-assessing sentences. We present StereoSet, a large-scale natural dataset in English to measure stereotypical biases in four domains: gender, profession, race, and religion. We evaluate popular models like BERT, GPT-2, RoBERTa, and XLNet on our dataset and show that these models exhibit strong stereotypical biases. We also present a leaderboard with a hidden test set to track the bias of future language models at https://stereoset.mit.edu

Key facts. Cited by 911 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 92; 2023: 200; 2024: 216; 2025: 194; 2026: 159.

911 papers.

Sort:

By area:

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  3. 2021Ethical and social risks of harm from Language Modelscited by 957
  4. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  5. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  6. 2023StarCoder: may the source be with you!cited by 790
  7. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  8. 2023Whose Opinions Do Language Models Reflect?cited by 759
  9. 2022Galactica: A Large Language Model for Sciencecited by 736
  10. 2023A Comprehensive Overview of Large Language Modelscited by 639
  11. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  12. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
38 more
  1. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  2. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  3. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  4. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  5. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 294
  6. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  7. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  8. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  9. 2022Prompting GPT-3 To Be Reliablecited by 258
  10. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  11. 2023Large Language Model Alignment: A Surveycited by 241
  12. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  13. 2023An Overview of Catastrophic AI Riskscited by 214
  14. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  15. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  16. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  17. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  18. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  19. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  20. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  21. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  22. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  23. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 171
  24. 2021Alignment of Language Agentscited by 167
  25. 2023A Categorical Archive of ChatGPT Failurescited by 167
  26. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  27. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  28. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  29. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  30. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  31. 2021RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Modelscited by 149
  32. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  33. 2023Debiasing Vision-Language Models via Biased Promptscited by 147
  34. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  35. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  36. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  37. 2023Auditing large language models: a three-layered approachcited by 135
  38. 2023EvoPrompting: Language Models for Code-Level Neural Architecture Searchcited by 129

50 most cited of 911 All 911 (JSON)