# Learning from Red Teaming: Gender Bias Provocation and Mitigation in Large Language Models

arXiv: https://arxiv.org/abs/2310.11079 · HTML: https://mlsanity.com/paper/2310.11079/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 17 Oct 2023.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Mitigating Unwanted Biases with Adversarial Learning](https://mlsanity.com/paper/1801.07593/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Models](https://mlsanity.com/paper/1610.02424/) (2016)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Gender Bias in Neural Natural Language Processing](https://mlsanity.com/paper/1807.11714/) (2018)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Towards Controllable Biases in Language Generation](https://mlsanity.com/paper/2005.00268/) (2020)

## Built on by (most cited first, 3 of 3)

- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- ['Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMs](https://mlsanity.com/paper/2409.13484/) (2024)
- [Bias in Text Embedding Models](https://mlsanity.com/paper/2406.12138/) (2024)

## Abstract (arXiv)

Recently, researchers have made considerable improvements in dialogue systems with the progress of large language models (LLMs) such as ChatGPT and GPT-4. These LLM-based chatbots encode the potential biases while retaining disparities that can harm humans during interactions. The traditional biases investigation methods often rely on human-written test cases. However, these test cases are usually expensive and limited. In this work, we propose a first-of-its-kind method that automatically generates test cases to detect LLMs' potential gender bias. We apply our method to three well-known LLMs and find that the generated test cases effectively identify the presence of biases. To address the biases identified, we propose a mitigation strategy that uses the generated test cases as demonstrations for in-context learning to circumvent the need for parameter fine-tuning. The experimental results show that LLMs generate fairer responses with the proposed approach.