Common to Whom? Regional Cultural Commonsense and LLM Bias in India
Not on the map. Why?
Abstract (arXiv)
Existing cultural commonsense benchmarks treat nations as monolithic, assuming uniform practices within national boundaries. But does cultural commonsense hold uniformly within a nation, or does it vary at the sub-national level? We introduce Indica, the first benchmark designed to test LLMs' ability to address this question, focusing on India - a nation of 28 states, 8 union territories, and 22 official languages. We collect human-annotated answers from five Indian regions (North, South, East, West, and Central) across 515 questions spanning 8 domains of everyday life, yielding 1,630 region-specific question-answer pairs. Strikingly, only 39.4% of questions elicit agreement across all five regions, demonstrating that cultural commonsense in India is predominantly regional, not national. We evaluate eight state-of-the-art LLMs and find two critical gaps: models achieve only 13.4%-20.9% accuracy on region-specific questions, and they exhibit geographic bias, over-selecting Central and North India as the "default" (selected 30-40% more often than expected) while under-representing East and West. Beyond India, our methodology provides a generalizable framework for evaluating cultural commonsense in any culturally heterogeneous nation, from question design grounded in anthropological taxonomy, to regional data collection, to bias measurement.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 19 of its 52 references that are in the map. First posted on arXiv 22 Jan 2026.
Builds on
19 of 52 references.
References · 19
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
- 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
- 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
- 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
- 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
- 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
- 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
- 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 50
- 2024IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Contextcited by 32
- 2024MILU: A Multi-task Indic Language Understanding Benchmarkcited by 26
7 more
- 2024DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subculturescited by 23
- 2025SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culturecited by 14
- 2024L3Cube-IndicQuest: A Benchmark Question Answering Dataset for Evaluating Knowledge of LLMs in Indic Contextcited by 10
- 2024Susu Box or Piggy Bank: Assessing Cultural Commonsense Knowledge between Ghana and the U.Scited by 9
- 2025IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understandingcited by 9
- 2025FairI Tales: Evaluation of Fairness in Indian Contexts with a Focus on Bias and Stereotypescited by 9
- 2025AMBEDKAR-A Multi-level Bias Elimination through a Decoding Approach with Knowledge Augmentation for Robust Constitutional Alignment of Language Modelscited by 2
Built on by
Citing papers per year: 2026: 3.
3 papers.