mlsanityData to 14 Oct 2026

Gender bias and stereotypes in Large Language Models

arXiv 2308.14921 ↗First posted 28 Aug 2023cs.CL cs.CY cs.LGcited by 187

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have made substantial progress in the past several months, shattering state-of-the-art benchmarks in many domains. This paper investigates LLMs' behavior with respect to gender stereotypes, a known issue for prior models. We use a simple paradigm to test the presence of gender bias, building on but differing from WinoBias, a commonly used gender bias dataset, which is likely to be included in the training data of current LLMs. We test four recently published LLMs and demonstrate that they express biased assumptions about men and women's occupations. Our contributions in this paper are as follows: (a) LLMs are 3-6 times more likely to choose an occupation that stereotypically aligns with a person's gender; (b) these choices align with people's perceptions better than with the ground truth as reflected in official job statistics; (c) LLMs in fact amplify the bias beyond what is reflected in perceptions or the ground truth; (d) LLMs ignore crucial ambiguities in sentence structure 95% of the time in our study items, but when explicitly prompted, they recognize the ambiguity; (e) LLMs provide explanations for their choices that are factually inaccurate and likely obscure the true reason behind their predictions. That is, they provide rationalizations of their biased behavior. This highlights a key property of these models: LLMs are trained on imbalanced datasets; as such, even with the recent successes of reinforcement learning with human feedback, they tend to reflect those imbalances back at us. As with other types of societal biases, we suggest that LLMs must be carefully tested to ensure that they treat minoritized individuals and communities equitably.

Key facts. Cited by 187 papers in the map (data through 14 Oct 2026). Builds on 40 of its 127 references that are in the map. First posted on arXiv 28 Aug 2023.

Builds on

40 of 127 references.

References · 40

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  4. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  5. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  6. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  7. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  8. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
  9. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  10. 2019Release Strategies and the Social Impacts of Language Modelscited by 513
  11. 2018Gender Bias in Coreference Resolutioncited by 442
  12. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
28 more
  1. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  2. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  3. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 380
  4. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  5. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 364
  6. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  7. 2019Measuring Bias in Contextualized Word Representationscited by 276
  8. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 253
  9. 2019Evaluating Gender Bias in Machine Translationcited by 249
  10. 2018Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systemscited by 234
  11. 2018Gender Bias in Neural Natural Language Processingcited by 211
  12. 2019Gender Bias in Contextualized Word Embeddingscited by 203
  13. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  14. 2018Learning Gender-Neutral Word Embeddingscited by 169
  15. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
  16. 2018Reducing Gender Bias in Abusive Language Detectioncited by 152
  17. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  18. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  19. 2022ChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reportscited by 101
  20. 2019Getting Gender Right in Neural Machine Translationcited by 98
  21. 2019Evaluating the Underlying Gender Bias in Contextualized Word Embeddingscited by 94
  22. 2023The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challengescited by 59
  23. 2023Nationality Bias in Text Generationcited by 50
  24. 2018Feature-Wise Bias Amplificationcited by 31
  25. 2018What are the biases in my word embedding?cited by 30
  26. 2020Mitigating Gender Bias Amplification in Distribution by Posterior Regularizationcited by 27
  27. 2022The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategiescited by 20
  28. 2023How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Studycited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 80; 2025: 77; 2026: 25.

187 papers.

Sort:

By area:

  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  2. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  4. 2024Rethinking Machine Unlearning for Large Language Modelscited by 257
  5. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  6. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  7. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  8. 2024Bias in Large Language Models: Origin, Evaluation, and Mitigationcited by 77
  9. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 74
  10. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  11. 2024(A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advicecited by 63
  12. 2024Fairness in Large Language Models: A Taxonomic Surveycited by 58
38 more
  1. 2024Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMscited by 55
  2. 2024What's in a Name? Auditing Large Language Models for Race and Gender Biascited by 51
  3. 2024BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Datacited by 50
  4. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  5. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 43
  6. 2025LLM Unlearning via Neural Activation Redirectioncited by 41
  7. 2023Assessing LLMs for Moral Value Pluralismcited by 34
  8. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 33
  9. 2024Network Formation and Dynamics Among Multi-LLMscited by 29
  10. 2024The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiringcited by 29
  11. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  12. 2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
  13. 2024AI-AI Bias: large language models favor communications generated by large language modelscited by 26
  14. 2024Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attackscited by 26
  15. 2024From Lived Experience to Insight: Unpacking the Psychological Risks of Using AI Conversational Agentscited by 25
  16. 2025Benchmarking Adversarial Robustness to Bias Elicitation in Large Language Models: Scalable Automated Assessment with LLM-as-a-Judgecited by 23
  17. 2025BLUR: A Bi-Level Optimization Approach for LLM Unlearningcited by 23
  18. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 22
  19. 2024ChatGPT as Research Scientist: Probing GPT's Capabilities as a Research Librarian, Research Ethicist, Data Generator and Data Predictorcited by 21
  20. 2024Listen and Speak Fairly: A Study on Semantic Gender Bias in Speech Integrated Large Language Modelscited by 21
  21. 2024Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigationcited by 17
  22. 2024When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settingscited by 17
  23. 2024Mix Data or Merge Models? Optimizing for Diverse Multi-Task Learningcited by 17
  24. 2025Unmasking Conversational Bias in AI Multiagent Systemscited by 17
  25. 2024On Catastrophic Inheritance of Large Foundation Modelscited by 15
  26. 2024LLM Whisperer: An Inconspicuous Attack to Bias LLM Responsescited by 15
  27. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
  28. 2025Position: Editing Large Language Models Poses Serious Safety Riskscited by 15
  29. 2025Dissecting Bias in LLMs: A Mechanistic Interpretability Perspectivecited by 15
  30. 2024An Evaluation of Estimative Uncertainty in Large Language Modelscited by 13
  31. 2024A Survey on Human-Centric LLMscited by 13
  32. 2024Fairness Definitions in Language Models Explainedcited by 11
  33. 2024Fairness Testing of Large Language Models in Role-Playingcited by 11
  34. 2024Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answeringcited by 11
  35. 2025Detecting Prefix Bias in LLM-based Reward Modelscited by 11
  36. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
  37. 2024A survey on fairness of large language models in e-commerce: progress, application, and challengecited by 10
  38. 2024Certifying Counterfactual Bias in LLMscited by 10

50 most cited of 187 All 187 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection
  2. 2024Protected group bias and stereotypes in Large Language Models
  3. 2025EuroGEST: Investigating gender stereotypes in multilingual language models
  4. 2024Disclosure and Mitigation of Gender Bias in LLMs
  5. 2024Interpreting Bias in Large Language Models: A Feature-Based Approach
  6. 2024Gender Bias in Decision-Making with Large Language Models: A Study of Relationship Conflicts
  7. 2024Bias and Volatility: A Statistical Framework for Evaluating Large Language Model's Stereotypes and the Associated Generation Inconsistency
  8. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models