# AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkani

arXiv: https://arxiv.org/abs/2606.15191 · HTML: https://mlsanity.com/paper/2606.15191/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 13 Jun 2026.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [MuRIL: Multilingual Representations for Indian Languages](https://mlsanity.com/paper/2103.10730/) (2021)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context](https://mlsanity.com/paper/2403.20147/) (2024)
- [Socially Aware Bias Measurements for Hindi Language Representations](https://mlsanity.com/paper/2110.07871/) (2021)
- [A Better Way to Do Masked Language Model Scoring](https://mlsanity.com/paper/2305.10588/) (2023)
- [Presumed Cultural Identity: How Names Shape LLM Responses](https://mlsanity.com/paper/2502.11995/) (2025)
- [AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings](https://mlsanity.com/paper/2011.01575/) (2020)
- [The Token Tax: Systematic Bias in Multilingual Tokenization](https://mlsanity.com/paper/2509.05486/) (2025)
- [Do Multilingual Large Language Models Mitigate Stereotype Bias?](https://mlsanity.com/paper/2407.05740/) (2024)
- [Attributing Culture-Conditioned Generations to Pretraining Corpora](https://mlsanity.com/paper/2412.20760/) (2024)
- [Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch](https://mlsanity.com/paper/2507.16442/) (2025)

## Built on by (most cited first, 1 of 1)

- [Status Association Does Not Reliably Predict Decision Leakage](https://mlsanity.com/paper/2608.10089/) (2026)

## Abstract (arXiv)

Socio-cultural stereotypical bias is an important consideration in the development and deployment of NLP systems. It is however often considered only at the national level, despite rich subnational socio-cultural structures. We present AmchiBias, the first benchmark for measuring socio-cultural stereotypical bias for the Indian state of Goa with its unique historically multicultural setting. It covers various Goan identity groups and comprises 313 minimal pairs across eight sociodemographic dimensions in both English and Devanagari Konkani. We then evaluate stereotypical bias in five multilingual encoder models on this benchmark. We find near-chance scores in Konkani, reflecting language incompetence for general multilingual models and a lack of Goan cultural competence for Indian language models. Queried in English, models with a stronger Indian language coverage show higher bias for pan-Indian groups than hyperlocal Goan groups. This suggests the English signal reflects pan-Indian pretraining associations rather than genuine Goan cultural knowledge. Our findings highlight a critical gap in low-resource multilingual NLP evaluation for hyperlocal community identities.