mlsanityData to 7 Oct 2026

Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluations

arXiv 2503.06987 ↗First posted 10 Mar 2025cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Measuring social bias in large language models (LLMs) is crucial, but existing bias evaluation methods struggle to assess bias in long-form generation. We propose a Bias Benchmark for Generation (BBG), an adaptation of the Bias Benchmark for QA (BBQ), designed to evaluate social bias in long-form generation by having LLMs generate continuations of story prompts. Building our benchmark in English and Korean, we measure the probability of neutral and biased generations across ten LLMs. We also compare our long-form story generation evaluation results with multiple-choice BBQ evaluation, showing that the two approaches produce inconsistent results.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 18 of its 33 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

18 of 33 references.

References · 18

  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2021Ethical and social risks of harm from Language Modelscited by 957
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  4. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  5. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  7. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  8. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  9. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
  10. 2024Can multiple-choice questions really be useful in detecting the abilities of LLMs?cited by 66
  11. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53
  12. 2023Nationality Bias in Text Generationcited by 50
6 more
  1. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 32
  2. 2024SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypescited by 24
  3. 2024"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversationscited by 24
  4. 2024Large Language Models Still Exhibit Bias in Long Textcited by 16
  5. 2024Evaluating Gender Bias of LLMs in Making Morality Judgementscited by 15
  6. 2022Towards Robust NLG Bias Evaluation with Syntactically-diverse Promptscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

  1. 2025RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivitycited by 4
  2. 2026Mi:dm 2.0 Korea-centric Bilingual Language Modelscited by 3
  3. 2026Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineeringcited by 3
  4. 2025EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answeringcited by 1
  5. 2025Measuring Mechanistic Independence: Can Bias Be Removed Without Erasing Demographics?cited by 1
  6. 2026Investigating Social Bias in Narrative Image Generationcited by 1
  7. 2026Effects of Answer Format Variation on Gender Bias in Large Language Modelscited by 1
  8. 2026Widespread Gender and Pronoun Bias in Moral Judgments Across LLMscited by 0
  9. 2026Intersectional Fairness in Large Language Modelscited by 0
  10. 2026GPF-LiveNews: A Streaming Evaluation Protocol for Group-Conditioned Framing in Large Language Modelscited by 0
  11. 2026VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputscited by 0
  12. 2026How Identity and Opinion Shape Political Sycophancy in LLMscited by 0
2 more
  1. 2026When Noise Fabricates Bias: The Fragility of LLM-as-a-Judge Bias Measurement under Noisy Textcited by 0
  2. 2026A Cross Community Agenda for Speech AIcited by 0