# Towards Understanding and Mitigating Social Biases in Language Models

arXiv: https://arxiv.org/abs/2106.13219 · HTML: https://mlsanity.com/paper/2106.13219/

## Key facts

- Cited by 296 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 57 references that are in the map.
- First posted on arXiv 24 Jun 2021.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Social Bias Frames: Reasoning about Social and Power Implications of Language](https://mlsanity.com/paper/1911.03891/) (2019)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Measuring Bias in Contextualized Word Representations](https://mlsanity.com/paper/1906.07337/) (2019)
- [Style Transfer from Non-Parallel Text by Cross-Alignment](https://mlsanity.com/paper/1705.09655/) (2017)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases](https://mlsanity.com/paper/2006.03955/) (2020)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [Assessing Social and Intersectional Biases in Contextualized Word Representations](https://mlsanity.com/paper/1911.01485/) (2019)

## Built on by (most cited first, 30 of 296)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study](https://mlsanity.com/paper/2404.10719/) (2024)
- [A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT](https://mlsanity.com/paper/2303.04226/) (2023)
- [Can Foundation Models Wrangle Your Data?](https://mlsanity.com/paper/2205.09911/) (2022)
- [Personality Traits in Large Language Models](https://mlsanity.com/paper/2307.00184/) (2023)
- [A Categorical Archive of ChatGPT Failures](https://mlsanity.com/paper/2302.03494/) (2023)
- [A Review on Language Models as Knowledge Bases](https://mlsanity.com/paper/2204.06031/) (2022)
- [Benchmark Data Contamination of Large Language Models: A Survey](https://mlsanity.com/paper/2406.04244/) (2024)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Large Language Models in Law: A Survey](https://mlsanity.com/paper/2312.03718/) (2023)
- [Almanac: Retrieval-Augmented Language Models for Clinical Medicine](https://mlsanity.com/paper/2303.01229/) (2023)
- [Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Models](https://mlsanity.com/paper/2307.14430/) (2023)
- [Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs](https://mlsanity.com/paper/2404.07103/) (2024)
- [Generative Language Models Exhibit Social Identity Biases](https://mlsanity.com/paper/2310.15819/) (2023)
- [Cognitive Bias in Decision-Making with LLMs](https://mlsanity.com/paper/2403.00811/) (2024)
- [Can LLM be a Personalized Judge?](https://mlsanity.com/paper/2406.11657/) (2024)
- [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://mlsanity.com/paper/2110.08527/) (2021)
- [Unifying Molecular and Textual Representations via Multi-task Language Modelling](https://mlsanity.com/paper/2301.12586/) (2023)
- [Large Language Models are Geographically Biased](https://mlsanity.com/paper/2402.02680/) (2024)
- [Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity](https://mlsanity.com/paper/2301.12867/) (2023)
- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)

## Abstract (arXiv)

As machine learning methods are deployed in real-world settings such as healthcare, legal systems, and social science, it is crucial to recognize how they shape social biases and stereotypes in these sensitive decision-making processes. Among such real-world deployments are large-scale pretrained language models (LMs) that can be potentially dangerous in manifesting undesirable representational biases - harmful biases resulting from stereotyping that propagate negative generalizations involving gender, race, religion, and other social constructs. As a step towards improving the fairness of LMs, we carefully define several sources of representational biases before proposing new benchmarks and metrics to measure them. With these tools, we propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information for high-fidelity text generation, thereby pushing forward the performance-fairness Pareto frontier.