mlsanityData to 7 Oct 2026

SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes

arXiv 2403.05696 ↗First posted 8 Mar 2024cs.CL cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

While generative multilingual models are rapidly being deployed, their safety and fairness evaluations are largely limited to resources collected in English. This is especially problematic for evaluations targeting inherently socio-cultural phenomena such as stereotyping, where it is important to build multi-lingual resources that reflect the stereotypes prevalent in respective language communities. However, gathering these resources, at scale, in varied languages and regions pose a significant challenge as it requires broad socio-cultural knowledge and can also be prohibitively expensive. To overcome this critical gap, we employ a recently introduced approach that couples LLM generations for scale with culturally situated validations for reliability, and build SeeGULL Multilingual, a global-scale multilingual dataset of social stereotypes, containing over 25K stereotypes, spanning 20 languages, with human annotations across 23 regions, and demonstrate its utility in identifying gaps in model evaluations. Content warning: Stereotypes shared in this paper can be offensive.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 17 of its 32 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

17 of 32 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  4. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  5. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  6. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  7. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  8. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  9. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  10. 2019On Measuring and Mitigating Biased Inferences of Word Embeddingscited by 112
  11. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 102
  12. 2023SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Modelscited by 66
5 more
  1. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  2. 2023On Evaluating and Mitigating Gender Biases in Multilingual Settingscited by 31
  3. 2021Socially Aware Bias Measurements for Hindi Language Representationscited by 29
  4. 2023Cross-Cultural Transfer Learning for Chinese Offensive Language Detectioncited by 21
  5. 2023SocialStigmaQA: A Benchmark to Uncover Stigma Amplification in Generative Language Modelscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 12; 2026: 6.

24 papers.

Sort:

By area:

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  3. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  4. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  5. 2024From Languages to Geographies: Towards Evaluating Cultural Bias in Hate Speech Datasetscited by 23
  6. 2025Social Bias Benchmark for Generation: A Comparison of Generation and QA-Based Evaluationscited by 14
  7. 2024Exploring Changes in Nation Perception with Nationality-Assigned Personas in LLMscited by 11
  8. 2025TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Storiescited by 10
  9. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  10. 2024JBBQ: Japanese Bias Benchmark for Analyzing Social Biases in Large Language Modelscited by 7
  11. 2026Computational Hermeneutics: Evaluating generative AI as a cultural technologycited by 7
  12. 2025HESEIA: A community-based dataset for evaluating social biases in large language models, co-designed in real school settings in Latin Americacited by 6
12 more
  1. 2025Towards Geo-Culturally Grounded LLM Generationscited by 5
  2. 2025BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Contextcited by 5
  3. 2025Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutchcited by 4
  4. 2024GeniL: A Multilingual Dataset on Generalizing Languagecited by 2
  5. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  6. 2025Social Bias in Multilingual Language Models: A Surveycited by 1
  7. 2026SAFARI: A Community-Engaged Approach and Dataset of Stereotype Resources in the Sub-Saharan African Contextcited by 1
  8. 2026Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaborationcited by 1
  9. 2025Text-to-Image Models and Their Representation of People from Different Nationalities Engaging in Activitiescited by 0
  10. 2026Exploratory As-Analyzed No-Detection of Culturally-Marked Predicate-Triggered PII Amplification in a Synthetic-English RAG Probe: A Predicate-Resource-Confounded Auditcited by 0
  11. 2026Tracing Stereotypes from Representation to Output in Multilingual LLMscited by 0
  12. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0