# "I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset

arXiv: https://arxiv.org/abs/2205.09209 · HTML: https://mlsanity.com/paper/2205.09209/

## Key facts

- Cited by 162 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 89 references that are in the map.
- First posted on arXiv 18 May 2022.

## Builds on (most cited first, 30 of 52)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Personalizing Dialogue Agents: I have a dog, do you have pets too?](https://mlsanity.com/paper/1801.07243/) (2018)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Recipes for building an open-domain chatbot](https://mlsanity.com/paper/2004.13637/) (2020)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Wizard of Wikipedia: Knowledge-Powered Conversational agents](https://mlsanity.com/paper/1811.01241/) (2018)
- [Towards a Human-like Open-Domain Chatbot](https://mlsanity.com/paper/2001.09977/) (2020)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [Measuring and Reducing Gendered Correlations in Pre-trained Models](https://mlsanity.com/paper/2010.06032/) (2020)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Societal Biases in Language Generation: Progress and Challenges](https://mlsanity.com/paper/2105.04054/) (2021)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [The Second Conversational Intelligence Challenge (ConvAI2)](https://mlsanity.com/paper/1902.00098/) (2019)

## Built on by (most cited first, 30 of 162)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [Gender bias and stereotypes in Large Language Models](https://mlsanity.com/paper/2308.14921/) (2023)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://mlsanity.com/paper/2305.14456/) (2023)
- [Spirit LM: Interleaved Spoken and Written Language Model](https://mlsanity.com/paper/2402.05755/) (2024)
- [A Survey on Fairness in Large Language Models](https://mlsanity.com/paper/2308.10149/) (2023)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Measuring Political Bias in Large Language Models: What Is Said and How It Is Said](https://mlsanity.com/paper/2403.18932/) (2024)
- [SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://mlsanity.com/paper/2404.05399/) (2024)
- [The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm](https://mlsanity.com/paper/2406.18682/) (2024)
- [Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems](https://mlsanity.com/paper/2401.05778/) (2024)
- [Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents](https://mlsanity.com/paper/2411.09523/) (2024)
- [TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Models](https://mlsanity.com/paper/2306.11507/) (2023)
- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [SocialCounterfactuals: Probing and Mitigating Intersectional Social Biases in Vision-Language Models with Counterfactual Examples](https://mlsanity.com/paper/2312.00825/) (2023)
- [Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks](https://mlsanity.com/paper/2306.05550/) (2023)
- [Supporting Human-AI Collaboration in Auditing LLMs with LLMs](https://mlsanity.com/paper/2304.09991/) (2023)
- [Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting](https://mlsanity.com/paper/2310.16523/) (2023)
- [WinoQueer: A Community-in-the-Loop Benchmark for Anti-LGBTQ+ Bias in Large Language Models](https://mlsanity.com/paper/2306.15087/) (2023)
- [Social Bias Evaluation for Large Language Models Requires Prompt Variations](https://mlsanity.com/paper/2407.03129/) (2024)
- [Are Personalized Stochastic Parrots More Dangerous? Evaluating Persona Biases in Dialogue Systems](https://mlsanity.com/paper/2310.05280/) (2023)
- [Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!](https://mlsanity.com/paper/2402.12343/) (2024)
- [FairMT-Bench: Benchmarking Fairness for Multi-turn Dialogue in Conversational LLMs](https://mlsanity.com/paper/2410.19317/) (2024)
- [MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs](https://mlsanity.com/paper/2406.07243/) (2024)

## Abstract (arXiv)

As language models grow in popularity, it becomes increasingly important to clearly measure all possible markers of demographic identity in order to avoid perpetuating existing societal harms. Many datasets for measuring bias currently exist, but they are restricted in their coverage of demographic axes and are commonly used with preset bias tests that presuppose which types of biases models can exhibit. In this work, we present a new, more inclusive bias measurement dataset, HolisticBias, which includes nearly 600 descriptor terms across 13 different demographic axes. HolisticBias was assembled in a participatory process including experts and community members with lived experience of these terms. These descriptors combine with a set of bias measurement templates to produce over 450,000 unique sentence prompts, which we use to explore, identify, and reduce novel forms of bias in several generative models. We demonstrate that HolisticBias is effective at measuring previously undetectable biases in token likelihoods from language models, as well as in an offensiveness classifier. We will invite additions and amendments to the dataset, which we hope will serve as a basis for more easy-to-use and standardized methods for evaluating bias in NLP models.