mlsanityData to 7 Oct 2026

MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs

arXiv 2406.07243 ↗First posted 11 Jun 2024cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Generative large language models (LLMs) have been shown to exhibit harmful biases and stereotypes. While safety fine-tuning typically takes place in English, if at all, these models are being used by speakers of many different languages. There is existing evidence that the performance of these models is inconsistent across languages and that they discriminate based on demographic factors of the user. Motivated by this, we investigate whether the social stereotypes exhibited by LLMs differ as a function of the language used to prompt them, while controlling for cultural differences and task accuracy. To this end, we present MBBQ (Multilingual Bias Benchmark for Question-answering), a carefully curated version of the English BBQ dataset extended to Dutch, Spanish, and Turkish, which measures stereotypes commonly held across these languages. We further complement MBBQ with a parallel control dataset to measure task performance on the question-answering task independently of bias. Our results based on several open-source and proprietary LLMs confirm that some non-English languages suffer from bias more than English, even when controlling for cultural shifts. Moreover, we observe significant cross-lingual differences in bias behaviour for all except the most accurate models. With the release of MBBQ, we hope to encourage further research on bias in multilingual settings. The dataset and code are available at https://github.com/Veranep/MBBQ.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 50 of its 66 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

50 of 66 references.

References · 50

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Mistral 7Bcited by 3,065
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  7. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  8. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  9. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  12. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
38 more
  1. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  2. 2023The Falcon Series of Open Language Modelscited by 526
  3. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  4. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  5. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  6. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  7. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  8. 2023SafetyBench: Evaluating the Safety of Large Language Modelscited by 189
  9. 2022"I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Datasetcited by 160
  10. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  11. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  12. 2022Large Language Models Meet NL2Code: A Surveycited by 135
  13. 2020UnQovering Stereotyping Biases via Underspecified Questionscited by 124
  14. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  15. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  16. 2019Does Gender Matter? Towards Fairness in Dialogue Systemscited by 89
  17. 2021On Measures of Biases and Harms in NLPcited by 81
  18. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  19. 2023KoBBQ: Korean Bias Benchmark for Question Answeringcited by 71
  20. 2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
  21. 2024Self-Debiasing Large Language Models: Zero-Shot Recognition and Reduction of Stereotypescited by 58
  22. 2023ROBBIE: Robust Bias Evaluation of Large Generative Language Modelscited by 53
  23. 2020Mitigating Gender Bias for Neural Dialogue Generation with Adversarial Learningcited by 44
  24. 2024Angry Men, Sad Women: Large Language Models Reflect Gendered Stereotypes in Emotion Attributioncited by 39
  25. 2023Comparing Biases and the Impact of Multilingual Training across Multiple Languagescited by 36
  26. 2024Gender Bias in Large Language Models across Multiple Languagescited by 35
  27. 2024Dialect prejudice predicts AI decisions about people's character, employability, and criminalitycited by 35
  28. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  29. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  30. 2023Global Voices, Local Biases: Socio-Cultural Prejudices across Languagescited by 23
  31. 2023CBBQ: A Chinese Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Modelscited by 22
  32. 2023Separating form and meaning: Using self-consistency to quantify task understanding across multiple sensescited by 15
  33. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  34. 2023Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniquescited by 14
  35. 2023Bias Beyond English: Counterfactual Tests for Bias in Sentiment Analysis in Four Languagescited by 8
  36. 2023Mitigating Bias for Question Answering Models by Tracking Bias Influencecited by 8
  37. 2023StereoMap: Quantifying the Awareness of Human-like Stereotypes in Large Language Modelscited by 6
  38. 2024The Gaps between Pre-train and Downstream Settings in Bias Evaluation and Debiasingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 16; 2026: 13.

32 papers.

Sort:

By area:

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2024Social Bias Evaluation for Large Language Models Requires Prompt Variationscited by 39
  3. 2025Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluationscited by 14
  4. 2025DAIQ: Auditing Demographic Attribute Inference from Question in LLMscited by 9
  5. 2025Deterministic or probabilistic? The psychology of LLMs as random number generatorscited by 6
  6. 2025Do Political Opinions Transfer Between Western Languages? An Analysis of Unaligned and Aligned Multilingual LLMscited by 6
  7. 2024Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigationcited by 5
  8. 2025Intent-Aware Self-Correction for Mitigating Social Biases in Large Language Modelscited by 5
  9. 2025BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Contextcited by 5
  10. 2025PakBBQ: A Culturally Adapted Bias Benchmark for QAcited by 4
  11. 2025Donors and Recipients: On Asymmetric Transfer Across Tasks and Languages with Parameter-Efficient Fine-Tuningcited by 4
  12. 2025Gradual Code-Switching as Inference-Time Cross-Lingual Representational Alignment for LLMscited by 3
20 more
  1. 2025Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Modelscited by 2
  2. 2025Bias Mitigation or Cultural Commonsense? Evaluating LLMs with a Japanese Datasetcited by 2
  3. 2026CAGE: A Framework for Culturally Adaptive Red-Teaming Benchmark Generationcited by 2
  4. 2025EsBBQ and CaBBQ: The Spanish and Catalan Bias Benchmarks for Question Answeringcited by 1
  5. 2025GG-BBQ: German Gender Bias Benchmark for Question Answeringcited by 1
  6. 2025Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Studycited by 1
  7. 2025Cross-Language Bias Examination in Large Language Modelscited by 1
  8. 2026TildeOpen LLM: Leveraging Curriculum Learning to Achieve Equitable Language Representationcited by 1
  9. 2026Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaborationcited by 1
  10. 2025Assessing Agentic Large Language Models in Multilingual National Biascited by 0
  11. 2026Preserving Fairness and Safety in Quantized LLMs Through Critical Weight Protectioncited by 0
  12. 2026SoS: Analysis of Surface over Semantics in Multilingual Text-To-Image Generationcited by 0
  13. 2026Toward Responsible and Epistemically Grounded Multilingual LLMs for Computational Social Science and Humanitiescited by 0
  14. 2026Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahilicited by 0
  15. 2026From Values to Benchmarks: Evaluating Large Language Models for Governmental Use in Dutchcited by 0
  16. 2026Exploratory As-Analyzed No-Detection of Culturally-Marked Predicate-Triggered PII Amplification in a Synthetic-English RAG Probe: A Predicate-Resource-Confounded Auditcited by 0
  17. 2026WinoQueer-NL: Assessing Bias in Dutch Language Models toward LGBTQ+ Identitiescited by 0
  18. 2026Tracing Stereotypes from Representation to Output in Multilingual LLMscited by 0
  19. 2026DyMT-ESB: Dynamic Multi-Turn Evaluation of Social Bias in User-LLM Interactionscited by 0
  20. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0