mlsanityData to 7 Oct 2026

Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models

arXiv 2408.03907 ↗First posted 7 Aug 2024cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have excelled at language understanding and generating human-level text. However, even with supervised training and human alignment, these LLMs are susceptible to adversarial attacks where malicious users can prompt the model to generate undesirable text. LLMs also inherently encode potential biases that can cause various harmful effects during interactions. Bias evaluation metrics lack standards as well as consensus and existing methods often rely on human-generated templates and annotations which are expensive and labor intensive. In this work, we train models to automatically create adversarial prompts to elicit biased responses from target LLMs. We present LLM- based bias evaluation metrics and also analyze several existing automatic evaluation methods and metrics. We analyze the various nuances of model responses, identify the strengths and weaknesses of model families, and assess where evaluation methods fall short. We compare these metrics to human evaluation and validate that the LLM-as-a-Judge metric aligns with human judgement on bias in response generation.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

27 of 39 references.

References · 27

  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2022Red Teaming Language Models with Language Modelscited by 983
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  4. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  5. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  6. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  7. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  8. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  9. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  10. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  11. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
15 more
  1. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  2. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  3. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
  4. 2019Evaluating Gender Bias in Machine Translationcited by 247
  5. 2018Gender Bias in Neural Natural Language Processingcited by 210
  6. 2019Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphologycited by 200
  7. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  8. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  9. 2018Learning Gender-Neutral Word Embeddingscited by 169
  10. 2023Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speechcited by 165
  11. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  12. 2019Evaluating the Underlying Gender Bias in Contextualized Word Embeddingscited by 94
  13. 2022On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representationscited by 75
  14. 2022Theories of "Gender" in NLP Bias Researchcited by 62
  15. 2022Challenges in Measuring Bias via Open-Ended Language Generationcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 20; 2026: 5.

33 papers.

Sort:

By area:

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  3. 2024FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMscited by 36
  4. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  5. 2025Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoningcited by 20
  6. 2026Quantifying Trust: Financial Risk Management for Trustworthy AI Agentscited by 8
  7. 2025Evaluating the Effect of Retrieval Augmentation on Social Biasescited by 7
  8. 2024Biased AI can Influence Political Decision-Makingcited by 6
  9. 2025Mitigating Bias in RAG: Controlling the Embeddercited by 6
  10. 2024The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Modelscited by 5
  11. 2025More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Modelscited by 5
  12. 2025FAIRE: Assessing Racial and Gender Bias in AI-Driven Resume Evaluationscited by 5
21 more
  1. 2025Neutralizing the Narrative: AI-Powered Debiasing of Online News Articlescited by 4
  2. 2025BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Modelscited by 4
  3. 2025Can Global XAI Methods Reveal Injected Behaviours in LLMs? SHAP vs Rule Extraction vs RuleSHAPcited by 3
  4. 2025Mental Health Equity in LLMs: Leveraging Multi-Hop Question Answering to Detect Amplified and Silenced Perspectivescited by 3
  5. 2024On the Role of Speech Data in Reducing Toxicity Detection Biascited by 2
  6. 2025DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Decoupled Reasoningcited by 2
  7. 2025LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Modelscited by 2
  8. 2024Gender Bias of LLM in Economics: An Existentialism Perspectivecited by 1
  9. 2025PinLanding: Content-First Keyword Landing Page Generation via Multi-Modal AI for Web-Scale Discoverycited by 1
  10. 2025The LLM Wears Prada: Analysing Gender Bias and Stereotypes through Online Shopping Datacited by 1
  11. 2025Toward Revealing Nuanced Biases in Medical LLMscited by 1
  12. 2025Do Biased Models Have Biased Thoughts?cited by 1
  13. 2025FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCscited by 1
  14. 2024Responsible AI in NLP: GUS-Net Span-Level Bias Detection Dataset and Benchmark for Generalizations, Unfairness, and Stereotypescited by 0
  15. 2025PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relationscited by 0
  16. 2025Enhancing Critical Thinking with AI: A Tailored Warning System for RAG Modelscited by 0
  17. 2025Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurementcited by 0
  18. 2025A Close Reading Approach to Gender Narrative Biases in AI-Generated Storiescited by 0
  19. 2026Analysis Of Linguistic Stereotypes in Single and Multi-Agent Generative AI Architecturescited by 0
  20. 2026Contaminated Collaboration: Measuring Gender Bias Transfer in LLM-Assisted Student Writingcited by 0
  21. 2026Identifying Implicit Bias in LLM-based Chat AI Toward People with Intellectual Disabilitiescited by 0