# Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models

arXiv: https://arxiv.org/abs/2408.03907 · HTML: https://mlsanity.com/paper/2408.03907/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 7 Aug 2024.

## Builds on (most cited first, 27 of 27)

- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models](https://mlsanity.com/paper/2306.11698/) (2023)
- [ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection](https://mlsanity.com/paper/2203.09509/) (2022)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- ["Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters](https://mlsanity.com/paper/2310.09219/) (2023)
- [Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks](https://mlsanity.com/paper/2310.10844/) (2023)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech](https://mlsanity.com/paper/2302.07736/) (2023)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019)
- [On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations](https://mlsanity.com/paper/2203.13928/) (2022)
- [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022)
- [Challenges in Measuring Bias via Open-Ended Language Generation](https://mlsanity.com/paper/2205.11601/) (2022)

## Built on by (most cited first, 30 of 33)

- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data](https://mlsanity.com/paper/2410.13341/) (2024)
- [FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs](https://mlsanity.com/paper/2410.19317/) (2024)
- [Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judges](https://mlsanity.com/paper/2601.20913/) (2026)
- [Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning](https://mlsanity.com/paper/2502.15361/) (2025)
- [Quantifying Trust: Financial Risk Management for Trustworthy AI Agents](https://mlsanity.com/paper/2604.03976/) (2026)
- [Evaluating the Effect of Retrieval Augmentation on Social Biases](https://mlsanity.com/paper/2502.17611/) (2025)
- [Biased AI can Influence Political Decision-Making](https://mlsanity.com/paper/2410.06415/) (2024)
- [Mitigating Bias in RAG: Controlling the Embedder](https://mlsanity.com/paper/2502.17390/) (2025)
- [The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models](https://mlsanity.com/paper/2411.03700/) (2024)
- [More Women, Same Stereotypes: Unpacking the Gender Bias Paradox in Large Language Models](https://mlsanity.com/paper/2503.15904/) (2025)
- [FAIRE: Assessing Racial and Gender Bias in AI-Driven Resume Evaluations](https://mlsanity.com/paper/2504.01420/) (2025)
- [Neutralizing the Narrative: AI-Powered Debiasing of Online News Articles](https://mlsanity.com/paper/2504.03520/) (2025)
- [BiasGuard: A Reasoning-enhanced Bias Detection Tool For Large Language Models](https://mlsanity.com/paper/2504.21299/) (2025)
- [Can Global XAI Methods Reveal Injected Behaviours in LLMs? SHAP vs Rule Extraction vs RuleSHAP](https://mlsanity.com/paper/2505.11189/) (2025)
- [Mental Health Equity in LLMs: Leveraging Multi-Hop Question Answering to Detect Amplified and Silenced Perspectives](https://mlsanity.com/paper/2506.18116/) (2025)
- [On the Role of Speech Data in Reducing Toxicity Detection Bias](https://mlsanity.com/paper/2411.08135/) (2024)
- [DR.GAP: Mitigating Bias in Large Language Models using Gender-Aware Prompting with Decoupled Reasoning](https://mlsanity.com/paper/2502.11603/) (2025)
- [LFTF: Locating First and Then Fine-Tuning for Mitigating Gender Bias in Large Language Models](https://mlsanity.com/paper/2505.15475/) (2025)
- [Gender Bias of LLM in Economics: An Existentialism Perspective](https://mlsanity.com/paper/2410.19775/) (2024)
- [PinLanding: Content-First Keyword Landing Page Generation via Multi-Modal AI for Web-Scale Discovery](https://mlsanity.com/paper/2503.00619/) (2025)
- [The LLM Wears Prada: Analysing Gender Bias and Stereotypes through Online Shopping Data](https://mlsanity.com/paper/2504.01951/) (2025)
- [Toward Revealing Nuanced Biases in Medical LLMs](https://mlsanity.com/paper/2507.21176/) (2025)
- [Do Biased Models Have Biased Thoughts?](https://mlsanity.com/paper/2508.06671/) (2025)
- [FAIRGAMER: Evaluating Social Biases in LLM-Based Video Game NPCs](https://mlsanity.com/paper/2508.17825/) (2025)
- [Responsible AI in NLP: GUS-Net Span-Level Bias Detection Dataset and Benchmark for Generalizations, Unfairness, and Stereotypes](https://mlsanity.com/paper/2410.08388/) (2024)
- [PCS: Perceived Confidence Scoring of Black Box LLMs with Metamorphic Relations](https://mlsanity.com/paper/2502.07186/) (2025)
- [Enhancing Critical Thinking with AI: A Tailored Warning System for RAG Models](https://mlsanity.com/paper/2504.16883/) (2025)
- [Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement](https://mlsanity.com/paper/2506.22372/) (2025)
- [A Close Reading Approach to Gender Narrative Biases in AI-Generated Stories](https://mlsanity.com/paper/2508.09651/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have excelled at language understanding and generating human-level text. However, even with supervised training and human alignment, these LLMs are susceptible to adversarial attacks where malicious users can prompt the model to generate undesirable text. LLMs also inherently encode potential biases that can cause various harmful effects during interactions. Bias evaluation metrics lack standards as well as consensus and existing methods often rely on human-generated templates and annotations which are expensive and labor intensive. In this work, we train models to automatically create adversarial prompts to elicit biased responses from target LLMs. We present LLM- based bias evaluation metrics and also analyze several existing automatic evaluation methods and metrics. We analyze the various nuances of model responses, identify the strengths and weaknesses of model families, and assess where evaluation methods fall short. We compare these metrics to human evaluation and validate that the LLM-as-a-Judge metric aligns with human judgement on bias in response generation.