# HESEIA: A community-based dataset for evaluating social biases in large language models, co-designed in real school settings in Latin America

arXiv: https://arxiv.org/abs/2505.24712 · HTML: https://mlsanity.com/paper/2505.24712/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 50 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 17 of 17)

- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems](https://mlsanity.com/paper/1805.04508/) (2018)
- [RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models](https://mlsanity.com/paper/2106.03521/) (2021)
- [Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings](https://mlsanity.com/paper/1904.04047/) (2019)
- [Power to the People? Opportunities and Challenges for Participatory AI](https://mlsanity.com/paper/2209.07572/) (2022)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022)
- [SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypes](https://mlsanity.com/paper/2403.05696/) (2024)
- [Position: Insights from Survey Methodology can Improve Training Data](https://mlsanity.com/paper/2403.01208/) (2024)
- [Measuring Intersectional Biases in Historical Documents](https://mlsanity.com/paper/2305.12376/) (2023)
- [Position: Cracking the Code of Cascading Disparity Towards Marginalized Communities](https://mlsanity.com/paper/2406.01757/) (2024)
- [What Makes An Expert? Reviewing How ML Researchers Define "Expert"](https://mlsanity.com/paper/2411.00179/) (2024)

## Built on by (most cited first, 6 of 6)

- [A Unified Framework to Quantify Cultural Intelligence of AI](https://mlsanity.com/paper/2603.01211/) (2026)
- [Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America](https://mlsanity.com/paper/2603.10001/) (2026)
- [Calibrated? Not for Everyone: How Sexual Orientation and Religious Markers Distort LLM Accuracy and Confidence in Medical QA](https://mlsanity.com/paper/2604.17316/) (2026)
- [SAFARI: A Community-Engaged Approach and Dataset of Stereotype Resources in the Sub-Saharan African Context](https://mlsanity.com/paper/2602.22404/) (2026)
- [Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025](https://mlsanity.com/paper/2606.02255/) (2026)
- [Toward Culturally Grounded Natural Language Processing](https://mlsanity.com/paper/2603.26013/) (2026)

## Abstract (arXiv)

Most resources for evaluating social biases in Large Language Models are developed without co-design from the communities affected by these biases, and rarely involve participatory approaches. We introduce HESEIA, a dataset of 46,499 sentences created in a professional development course. The course involved 370 high-school teachers and 5,370 students from 189 Latin-American schools. Unlike existing benchmarks, HESEIA captures intersectional biases across multiple demographic axes and school subjects. It reflects local contexts through the lived experience and pedagogical expertise of educators. Teachers used minimal pairs to create sentences that express stereotypes relevant to their school subjects and communities. We show the dataset diversity in term of demographic axes represented and also in terms of the knowledge areas included. We demonstrate that the dataset contains more stereotypes unrecognized by current LLMs than previous datasets. HESEIA is available to support bias assessments grounded in educational communities.