# Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings

arXiv: https://arxiv.org/abs/2412.06134 · HTML: https://mlsanity.com/paper/2412.06134/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 9 Dec 2024.

## Builds on (most cited first, 30 of 30)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [On Measuring Social Biases in Sentence Encoders](https://mlsanity.com/paper/1903.10561/) (2019)
- [Towards Understanding and Mitigating Social Biases in Language Models](https://mlsanity.com/paper/2106.13219/) (2021)
- [PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://mlsanity.com/paper/2406.15513/) (2024)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [UnQovering Stereotyping Biases via Underspecified Questions](https://mlsanity.com/paper/2010.02428/) (2020)
- [Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypes](https://mlsanity.com/paper/2402.01981/) (2024)
- [ROBBIE: Robust Bias Evaluation of Large Generative Language Models](https://mlsanity.com/paper/2311.18140/) (2023)
- [Evaluating Gender Bias in Large Language Models via Chain-of-Thought Prompting](https://mlsanity.com/paper/2401.15585/) (2024)
- [Mitigating Gender Bias in Distilled Language Models via Counterfactual Role Reversal](https://mlsanity.com/paper/2203.12574/) (2022)
- [LLM-Ensemble: Optimal Large Language Model Ensemble Method for E-commerce Product Attribute Value Extraction](https://mlsanity.com/paper/2403.00863/) (2024)
- [Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models](https://mlsanity.com/paper/2309.08902/) (2023)
- [Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment](https://mlsanity.com/paper/2403.02738/) (2024)
- [PREFER: Prompt Ensemble Learning via Feedback-Reflect-Refine](https://mlsanity.com/paper/2308.12033/) (2023)
- [Intersectional Bias in Causal Language Models](https://mlsanity.com/paper/2107.07691/) (2021)
- [A Robust Bias Mitigation Procedure Based on the Stereotype Content Model](https://mlsanity.com/paper/2210.14552/) (2022)
- [Gender Biases and Where to Find Them: Exploring Gender Bias in Pre-Trained Transformer-based Language Models Using Movement Pruning](https://mlsanity.com/paper/2207.02463/) (2022)
- [Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification](https://mlsanity.com/paper/2402.18502/) (2024)
- [AXOLOTL: Fairness through Assisted Self-Debiasing of Large Language Model Outputs](https://mlsanity.com/paper/2403.00198/) (2024)
- [Does Debiasing Inevitably Degrade the Model Performance](https://mlsanity.com/paper/2211.07350/) (2022)

## Built on by (most cited first, 10 of 10)

- [Does Reasoning Introduce Bias? A Study of Social Bias Evaluation and Mitigation in LLM Reasoning](https://mlsanity.com/paper/2502.15361/) (2025)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [BiasCause: Evaluate Socially Biased Causal Reasoning of Large Language Models](https://mlsanity.com/paper/2504.07997/) (2025)
- [Phare: A Safety Probe for Large Language Models](https://mlsanity.com/paper/2505.11365/) (2025)
- [Aligned Agents, Biased Swarm: Measuring Bias Amplification in Multi-Agent Systems](https://mlsanity.com/paper/2604.08963/) (2026)
- [QuantiBias: Benchmarking Quantization-Induced Bias in LLMs](https://mlsanity.com/paper/2607.21063/) (2026)
- [AfriStereo: A Culturally Grounded Dataset for Evaluating Stereotypical Bias in Large Language Models](https://mlsanity.com/paper/2511.22016/) (2025)
- [Intersectional Fairness in Large Language Models](https://mlsanity.com/paper/2604.20677/) (2026)
- [StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs](https://mlsanity.com/paper/2605.10442/) (2026)
- [Fairness Pruning: Locating Demographic Bias in GLU-MLP Layers via Differential Activations](https://mlsanity.com/paper/2607.28319/) (2026)

## Abstract (arXiv)

Current social bias benchmarks for Large Language Models (LLMs) primarily rely on predefined question formats like multiple-choice, limiting their ability to reflect the complexity and open-ended nature of real-world interactions. To close this gap, we extend an existing dataset BBQ (Parrish et al., 2022) to Open-BBQ, a comprehensive framework to evaluate the social bias of LLMs in open-ended settings by incorporating two additional question categories: fill-in-the-blank and short-answer. Since our new Open-BBQ dataset contains a lot of open-ended responses like sentences and paragraphs, we developed an evaluation process to detect biases from open-ended content by labeling sentences and paragraphs. In addition to this, we also found that existing debiasing methods, such as self-debiasing (Gallegos et al., 2024), have over-correction issues, which make the original correct answers incorrect. In order to solve this issue, we propose Composite Prompting, an In-context Learning (ICL) method combining structured examples with explicit chain-of-thought reasoning to form a unified instruction template for LLMs to explicitly identify content that needs debiasing. Experimental results show that the proposed method significantly reduces the bias for both GPT-3.5 and GPT-4o while maintaining high accuracy.