# BanStereoSet: A Dataset to Measure Stereotypical Social Biases in LLMs for Bangla

arXiv: https://arxiv.org/abs/2409.11638 · HTML: https://mlsanity.com/paper/2409.11638/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 18 Sep 2024.

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [Investigating Subtler Biases in LLMs: Ageism, Beauty, Institutional, and Nationality Bias in Generative Models](https://mlsanity.com/paper/2309.08902/) (2023)
- [IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context](https://mlsanity.com/paper/2403.20147/) (2024)
- [Global Voices, Local Biases: Socio-Cultural Prejudices across Languages](https://mlsanity.com/paper/2310.17586/) (2023)
- [AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings](https://mlsanity.com/paper/2011.01575/) (2020)
- ["Global is Good, Local is Bad?": Understanding Brand Bias in LLMs](https://mlsanity.com/paper/2406.13997/) (2024)
- [Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Bias](https://mlsanity.com/paper/2407.03536/) (2024)

## Built on by (most cited first, 3 of 3)

- [Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs](https://mlsanity.com/paper/2507.05686/) (2025)
- [Evaluating Regional Bias in LLMs From Abstract Stereotype to Concrete Social Decision-Making](https://mlsanity.com/paper/2607.27022/) (2026)
- [Status Association Does Not Reliably Predict Decision Leakage](https://mlsanity.com/paper/2608.10089/) (2026)

## Abstract (arXiv)

This study presents BanStereoSet, a dataset designed to evaluate stereotypical social biases in multilingual LLMs for the Bangla language. In an effort to extend the focus of bias research beyond English-centric datasets, we have localized the content from the StereoSet, IndiBias, and Kamruzzaman et. al.'s datasets, producing a resource tailored to capture biases prevalent within the Bangla-speaking community. Our BanStereoSet dataset consists of 1,194 sentences spanning 9 categories of bias: race, profession, gender, ageism, beauty, beauty in profession, region, caste, and religion. This dataset not only serves as a crucial tool for measuring bias in multilingual LLMs but also facilitates the exploration of stereotypical bias across different social categories, potentially guiding the development of more equitable language technologies in Bangladeshi contexts. Our analysis of several language models using this dataset indicates significant biases, reinforcing the necessity for culturally and linguistically adapted datasets to develop more equitable language technologies.