mlsanityData to 7 Oct 2026

IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context

arXiv 2403.20147 ↗First posted 29 Mar 2024cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

The pervasive influence of social biases in language data has sparked the need for benchmark datasets that capture and evaluate these biases in Large Language Models (LLMs). Existing efforts predominantly focus on English language and the Western context, leaving a void for a reliable dataset that encapsulates India's unique socio-cultural nuances. To bridge this gap, we introduce IndiBias, a comprehensive benchmarking dataset designed specifically for evaluating social biases in the Indian context. We filter and translate the existing CrowS-Pairs dataset to create a benchmark dataset suited to the Indian context in Hindi language. Additionally, we leverage LLMs including ChatGPT and InstructGPT to augment our dataset with diverse societal biases and stereotypes prevalent in India. The included bias dimensions encompass gender, religion, caste, age, region, physical appearance, and occupation. We also build a resource to address intersectional biases along three intersectional dimensions. Our dataset contains 800 sentence pairs and 300 tuples for bias measurement across different demographics. The dataset is available in English and Hindi, providing a size comparable to existing benchmark datasets. Furthermore, using IndiBias we compare ten different language models on multiple bias measurement metrics. We observed that the language models exhibit more bias across a majority of the intersectional groups.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 30 of its 59 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

30 of 59 references.

References · 30

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  6. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,290
  7. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  10. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  11. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  12. 2019On Measuring Social Biases in Sentence Encoderscited by 397
18 more
  1. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  2. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  3. 2021MuRIL: Multilingual Representations for Indian Languagescited by 177
  4. 2019Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddingscited by 146
  5. 2022mGPT: Few-Shot Learners Go Multilingualcited by 141
  6. 2019Assessing Social and Intersectional Biases in Contextualized Word Representationscited by 134
  7. 2021Gender Bias in Machine Translationcited by 116
  8. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 102
  9. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  10. 2020Hate Speech Detection and Racial Bias Mitigation in Social Media based on BERT modelcited by 87
  11. 2023SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Modelscited by 66
  12. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  13. 2021Socially Aware Bias Measurements for Hindi Language Representationscited by 29
  14. 2020Mitigating Gender Bias Amplification in Distribution by Posterior Regularizationcited by 27
  15. 2020AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddingscited by 20
  16. 2022Detecting Unintended Social Bias in Toxic Language Datasetscited by 12
  17. 2019Smart, Responsible, and Upper Caste Only: Measuring Caste Attitudes through Large-Scale Analysis of Matrimonial Profilescited by 5
  18. 2022Hollywood Identity Bias Dataset: A Context Oriented Bias Analysis of Movie Dialoguescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 15; 2026: 12.

32 papers.

Sort:

By area:

  1. 2025Measuring what Matters: Construct Validity in Large Language Model Benchmarkscited by 102
  2. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  3. 2024Social Bias in Large Language Models For Bangla: An Empirical Study on Gender and Religious Biascited by 13
  4. 2025Shifting Perspectives: Steering Vectors for Robust Bias Mitigation in LLMscited by 13
  5. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  6. 2025DECASTE: Unveiling Caste Stereotypes in Large Language Models through Multi-Dimensional Bias Analysiscited by 12
  7. 2025FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypescited by 9
  8. 2024KULTURE Bench: A Benchmark for Assessing Language Model in Korean Cultural Contextcited by 8
  9. 2025BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Contextcited by 5
  10. 2024BanStereoSet: A Dataset to Measure Stereotypical Social Biases in LLMs for Banglacited by 3
  11. 2025McBE: A Multi-task Chinese Bias Evaluation Benchmark for Large Language Modelscited by 3
  12. 2026Common to Whom? Regional Cultural Commonsense and LLM Bias in Indiacited by 3
20 more
  1. 2025Purdah and Patriarchy: Evaluating and Mitigating South Asian Biases in Open-Ended Multilingual LLM Generationscited by 2
  2. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  3. 2025Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMscited by 2
  4. 2024Who is better at math, Jenny or Jingzhen? Uncovering Stereotypes in Large Language Modelscited by 1
  5. 2025From n-gram to Attention: How Model Architectures Learn and Propagate Bias in Language Modelingcited by 1
  6. 2025NEAT: Concept driven Neuron Attribution in LLMscited by 1
  7. 2025Social Bias in Multilingual Language Models: A Surveycited by 1
  8. 2025IndiCASA: A Dataset and Bias Evaluation Framework in LLMs Using Contrastive Embedding Similarity in the Indian Contextcited by 1
  9. 2026AmchiBias: Measuring Stereotypical Bias in Goan Identity Groups with a Minimal Pair Dataset in English and Konkanicited by 1
  10. 2025Stereotype Detection as a Catalyst for Enhanced Bias Detection: A Multi-Task Learning Approachcited by 0
  11. 2026IndicFairFace: Balanced Indian Face Dataset for Auditing and Mitigating Geographical Bias in Vision-Language Modelscited by 0
  12. 2026Analysing LLM Persona Generation and Fairness Interpretation in Polarised Geopolitical Contextscited by 0
  13. 2026Invisible Influences: Investigating Implicit Intersectional Biases through Persona Engineering in Large Language Modelscited by 0
  14. 2026Safety Alignment Illusion: The Cross-Lingual Safety Gap in LLMscited by 0
  15. 2026Which India Survives Translation? Narrative Homogenisation Across Indian Oral Traditions in LLMscited by 0
  16. 2026LLMs Mirror Country-Specific Gender Patterns If Asked, but Skew Male When Generating Media in Local Languagescited by 0
  17. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0
  18. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0
  19. 2026RupeeBias: Auditing Demographic Bias in Indian Economic Guidance from Large Language Modelscited by 0
  20. 2026Acmite: Mitigating Gender Bias in LLMs through Concept-Guided Mutual Informationcited by 0