# Scalable and Culturally Specific Stereotype Dataset Construction via Human-LLM Collaboration

arXiv: https://arxiv.org/abs/2607.07895 · HTML: https://mlsanity.com/paper/2607.07895/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 44 references that are in the map.
- First posted on arXiv 8 Jul 2026.

## Builds on (most cited first, 14 of 14)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [Spanish Pre-trained BERT Model and Evaluation Data](https://mlsanity.com/paper/2308.02976/) (2023)
- [SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models](https://mlsanity.com/paper/2305.11840/) (2023)
- [Comparing Biases and the Impact of Multilingual Training across Multiple Languages](https://mlsanity.com/paper/2305.11242/) (2023)
- [MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMs](https://mlsanity.com/paper/2406.07243/) (2024)
- [On Evaluating and Mitigating Gender Biases in Multilingual Settings](https://mlsanity.com/paper/2307.01503/) (2023)
- [SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes](https://mlsanity.com/paper/2403.05696/) (2024)
- [Investigating Bias in Multilingual Language Models: Cross-Lingual Transfer of Debiasing Techniques](https://mlsanity.com/paper/2310.10310/) (2023)
- [How Different Is Stereotypical Bias Across Languages?](https://mlsanity.com/paper/2307.07331/) (2023)
- [A methodology to characterize bias and harmful stereotypes in natural language processing in Latin America](https://mlsanity.com/paper/2207.06591/) (2022)

## Built on by (most cited first, 1 of 1)

- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

Research on stereotypes in large language models (LLMs) has largely focused on English-speaking contexts, due to the lack of datasets in other languages and the high cost of manual annotation in underrepresented cultures. To address this gap, we introduce a cost-efficient human-LLM collaborative annotation framework and apply it to construct EspanStereo, a Spanish-language stereotype dataset spanning multiple Spanish-speaking countries across Europe and Latin America. EspanStereo captures both well-documented stereotypes from prior literature and culturally specific biases absent from English-centric resources. Using LLMs to generate candidate stereotypes and in-culture annotators to validate them, we demonstrate the framework's effectiveness in identifying nuanced, region-specific biases. Our evaluation of Spanish-supporting LLMs using EspanStereo reveals significant variation in stereotypical behavior across countries, highlighting the need for more culturally grounded assessments. Beyond Spanish, our framework is adaptable to other languages and regions, offering a scalable path toward multilingual stereotype benchmarks. This work broadens the scope of stereotype analysis in LLMs and lays the groundwork for comprehensive cross-cultural bias evaluation.