# Social Bias Evaluation for Large Language Models Requires Prompt Variations

arXiv: https://arxiv.org/abs/2407.03129 · HTML: https://mlsanity.com/paper/2407.03129/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 66 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 30 of 48)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [State of What Art? A Call for Multi-Prompt LLM Evaluation](https://mlsanity.com/paper/2401.00595/) (2023)
- [Prompting GPT-3 To Be Reliable](https://mlsanity.com/paper/2210.09150/) (2022)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Leveraging Large Language Models for Multiple Choice Question Answering](https://mlsanity.com/paper/2210.12353/) (2022)
- [Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology](https://mlsanity.com/paper/1906.04571/) (2019)
- [On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning](https://mlsanity.com/paper/2212.08061/) (2022)
- ["I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://mlsanity.com/paper/2205.09209/) (2022)
- [Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models](https://mlsanity.com/paper/2102.04130/) (2021)
- [UnQovering Stereotyping Biases via Underspecified Questions](https://mlsanity.com/paper/2010.02428/) (2020)

## Built on by (most cited first, 30 of 40)

- [Revealing and Reducing Gender Biases in Vision and Language Assistants (VLAs)](https://mlsanity.com/paper/2410.19314/) (2024)
- [Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMs](https://mlsanity.com/paper/2503.05371/) (2025)
- [A Large Scale Analysis of Gender Biases in Text-to-Image Generative Models](https://mlsanity.com/paper/2503.23398/) (2025)
- [AGR: Age Group fairness Reward for Bias Mitigation in LLMs](https://mlsanity.com/paper/2409.04340/) (2024)
- [ReliableEval: A Recipe for Stochastic LLM Evaluation via Method of Moments](https://mlsanity.com/paper/2505.22169/) (2025)
- [Paraphrasing Adversarial Attack on LLM-as-a-Reviewer](https://mlsanity.com/paper/2601.06884/) (2026)
- [VoxSafeBench: Not Just What Is Said, but Who, How, and Where](https://mlsanity.com/paper/2604.14548/) (2026)
- [The Personalization Trap: How User Memory Alters Emotional Reasoning in LLMs](https://mlsanity.com/paper/2510.09905/) (2025)
- [Awes, Laws, and Flaws From Today's LLM Research](https://mlsanity.com/paper/2408.15409/) (2024)
- [Intent-Aware Self-Correction for Mitigating Social Biases in Large Language Models](https://mlsanity.com/paper/2503.06011/) (2025)
- [The Thin Line Between Comprehension and Persuasion in LLMs](https://mlsanity.com/paper/2507.01936/) (2025)
- [Mind the Language Gap: Automated and Augmented Evaluation of Bias in LLMs for High- and Low-Resource Languages](https://mlsanity.com/paper/2504.18560/) (2025)
- [Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework](https://mlsanity.com/paper/2505.03563/) (2025)
- [Adaptive Prompting: Ad-hoc Prompt Composition for Social Bias Detection](https://mlsanity.com/paper/2502.06487/) (2025)
- [DIF: A Framework for Benchmarking and Verifying Implicit Bias in LLMs](https://mlsanity.com/paper/2505.10013/) (2025)
- [Evaluating Large Language Models (LLMs) in Financial NLP: A Comparative Study on Financial Report Analysis](https://mlsanity.com/paper/2507.22936/) (2025)
- [Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Dataset](https://mlsanity.com/paper/2509.24468/) (2025)
- [Syntactic Framing Fragility: An Audit of Robustness in LLM Ethical Decisions](https://mlsanity.com/paper/2601.09724/) (2025)
- [Language Shapes Mental Health Evaluations in Large Language Models](https://mlsanity.com/paper/2603.06910/) (2026)
- [Advertising in AI systems: Society must be vigilant](https://mlsanity.com/paper/2505.18425/) (2025)
- [What LLMs Think When You Don't Tell Them What to Think About?](https://mlsanity.com/paper/2602.01689/) (2026)
- [DeFrame: Debiasing Large Language Models Against Framing Effects](https://mlsanity.com/paper/2602.04306/) (2026)
- [The Impact of Ideological Discourses in RAG: A Case Study with COVID-19 Treatments](https://mlsanity.com/paper/2603.14838/) (2026)
- [JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs](https://mlsanity.com/paper/2603.20581/) (2026)
- [Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communities](https://mlsanity.com/paper/2605.04127/) (2026)
- [BiAxisBias: Evaluating LLM Bias Beyond a Single Prompt and a Single Explanation](https://mlsanity.com/paper/2605.09041/) (2026)
- [Effects of Answer Format Variation on Gender Bias in Large Language Models](https://mlsanity.com/paper/2608.17516/) (2026)
- [Token-Level Entropy Reveals Demographic Disparities in Large Language Models](https://mlsanity.com/paper/2501.19337/) (2025)
- [Balanced Multi-Factor In-Context Learning for Multilingual Large Language Models](https://mlsanity.com/paper/2502.11495/) (2025)
- [BIPOLAR: Polarization-based granular framework for LLM bias evaluation](https://mlsanity.com/paper/2508.11061/) (2025)

## Abstract (arXiv)

Warning: This paper contains examples of stereotypes and biases. Large Language Models (LLMs) exhibit considerable social biases, and various studies have tried to evaluate and mitigate these biases accurately. Previous studies use downstream tasks as prompts to examine the degree of social biases for evaluation and mitigation. While LLMs' output highly depends on prompts, previous studies evaluating and mitigating bias have often relied on a limited variety of prompts. In this paper, we investigate the sensitivity of LLMs when changing prompt variations (task instruction and prompt, few-shot examples, debias-prompt) by analyzing task performance and social bias of LLMs. Our experimental results reveal that LLMs are highly sensitive to prompts to the extent that the ranking of LLMs fluctuates when comparing models for task performance and social bias. Additionally, we show that LLMs have tradeoffs between performance and social bias caused by the prompts. Less bias from prompt setting may result in reduced performance. Moreover, the ambiguity of instances is one of the reasons for this sensitivity to prompts in advanced LLMs, leading to various outputs. We recommend using diverse prompts, as in this study, to compare the effects of prompts on social bias in LLMs.