EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answering
Not on the map. Why?
Abstract (arXiv)
Previous literature has largely shown that Large Language Models (LLMs) perpetuate social biases learnt from their pre-training data. Given the notable lack of resources for social bias evaluation in languages other than English, and for social contexts outside of the United States, this paper introduces the Spanish and the Catalan Bias Benchmarks for Question Answering (EsBBQ and CaBBQ). Based on the original BBQ, these two parallel datasets are designed to assess social bias across 10 categories using a multiple-choice QA setting, now adapted to the Spanish and Catalan languages and to the social context of Spain. We report evaluation results on different LLMs, factoring in model family, size and variant. Our results show that models tend to fail to choose the correct answer in ambiguous scenarios, and that high QA accuracy often correlates with greater reliance on social biases.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 59 references that are in the map. First posted on arXiv 15 Jul 2025.
Builds on
30 of 59 references.
References · 30
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
- 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
- 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
- 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
- 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
- 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
- 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 659
- 2018Gender Bias in Coreference Resolutioncited by 442
18 more
- 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 412
- 2019On Measuring Social Biases in Sentence Encoderscited by 398
- 2019Measuring Bias in Contextualized Word Representationscited by 276
- 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
- 2020Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biasescited by 167
- 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 162
- 2020Intrinsic Bias Metrics Do Not Correlate with Application Biascited by 148
- 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 135
- 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 126
- 2024EuroLLM: Multilingual Language Models for Europecited by 106
- 2021On Measures of Biases and Harms in NLPcited by 81
- 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
- 2022Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Taskscited by 48
- 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 35
- 2022The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarkscited by 31
- 2025Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluationscited by 16
- 2023This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Modelscited by 9
- 2023In-Depth Look at Word Filling Societal Bias Measurescited by 8
Built on by
Citing papers per year: 2026: 1.
1 paper.