# Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations

arXiv: https://arxiv.org/abs/2503.06987 · HTML: https://mlsanity.com/paper/2503.06987/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 10 Mar 2025.

## Builds on (most cited first, 18 of 18)

- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Bias and Fairness in Large Language Models: A Survey](https://mlsanity.com/paper/2309.00770/) (2023)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation](https://mlsanity.com/paper/2101.11718/) (2021)
- [Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models](https://mlsanity.com/paper/2305.18189/) (2023)
- [KoBBQ: Korean Bias Benchmark for Question Answering](https://mlsanity.com/paper/2307.16778/) (2023)
- [Can multiple-choice questions really be useful in detecting the abilities of LLMs?](https://mlsanity.com/paper/2403.17752/) (2024)
- [ROBBIE: Robust Bias Evaluation of Large Generative Language Models](https://mlsanity.com/paper/2311.18140/) (2023)
- [Nationality Bias in Text Generation](https://mlsanity.com/paper/2302.02463/) (2023)
- [MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs](https://mlsanity.com/paper/2406.07243/) (2024)
- ["They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversations](https://mlsanity.com/paper/2405.05378/) (2024)
- [SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes](https://mlsanity.com/paper/2403.05696/) (2024)
- [Large Language Models Still Exhibit Bias in Long Text](https://mlsanity.com/paper/2410.17519/) (2024)
- [Evaluating Gender Bias of LLMs in Making Morality Judgements](https://mlsanity.com/paper/2410.09992/) (2024)
- [Towards Robust NLG Bias Evaluation with Syntactically-diverse Prompts](https://mlsanity.com/paper/2212.01700/) (2022)

## Built on by (most cited first, 16 of 16)

- [Mi:dm 2.0 Korea-centric Bilingual Language Models](https://mlsanity.com/paper/2601.09066/) (2026)
- [RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity](https://mlsanity.com/paper/2509.25897/) (2025)
- [Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering](https://mlsanity.com/paper/2604.16790/) (2026)
- [EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering](https://mlsanity.com/paper/2507.11216/) (2025)
- [Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?](https://mlsanity.com/paper/2512.20796/) (2025)
- [QuantiBias: Benchmarking Quantization-Induced Bias in LLMs](https://mlsanity.com/paper/2607.21063/) (2026)
- [Investigating Social Bias in Narrative Image Generation](https://mlsanity.com/paper/2608.01780/) (2026)
- [Effects of Answer Format Variation on Gender Bias in Large Language Models](https://mlsanity.com/paper/2608.17516/) (2026)
- [Widespread Gender and Pronoun Bias in Moral Judgments Across LLMs](https://mlsanity.com/paper/2603.13636/) (2026)
- [Intersectional Fairness in Large Language Models](https://mlsanity.com/paper/2604.20677/) (2026)
- [GPF-LiveNews: A Streaming Evaluation Protocol for Group-Conditioned Framing in Large Language Models](https://mlsanity.com/paper/2605.28848/) (2026)
- [VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs](https://mlsanity.com/paper/2608.03810/) (2026)
- [How Identity and Opinion Shape Political Sycophancy in LLMs](https://mlsanity.com/paper/2608.29198/) (2026)
- [When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Text](https://mlsanity.com/paper/2609.11067/) (2026)
- [A Cross Community Agenda for Speech AI](https://mlsanity.com/paper/2609.13168/) (2026)
- [Noise Out, Bias In: Targeted Bias Injection in Diffusion Language Models via Closed-Loop Activation Steering](https://mlsanity.com/paper/2610.05894/) (2026)

## Abstract (arXiv)

Measuring social bias in large language models (LLMs) is crucial, but existing bias evaluation methods struggle to assess bias in long-form generation. We propose a Bias Benchmark for Generation (BBG), an adaptation of the Bias Benchmark for QA (BBQ), designed to evaluate social bias in long-form generation by having LLMs generate continuations of story prompts. Building our benchmark in English and Korean, we measure the probability of neutral and biased generations across ten LLMs. We also compare our long-form story generation evaluation results with multiple-choice BBQ evaluation, showing that the two approaches produce inconsistent results.