# DOSA: A Dataset of Social Artifacts from Different Indian Geographical Subcultures

arXiv: https://arxiv.org/abs/2403.14651 · HTML: https://mlsanity.com/paper/2403.14651/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 23 Feb 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 17 of 17)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do](https://mlsanity.com/paper/2103.11790/) (2021)
- [Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models](https://mlsanity.com/paper/2102.02503/) (2021)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [Extracting Cultural Commonsense Knowledge at Scale](https://mlsanity.com/paper/2210.07763/) (2022)
- [AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia](https://mlsanity.com/paper/2305.11844/) (2023)
- [Evaluating GPT-3 Generated Explanations for Hateful Content Moderation](https://mlsanity.com/paper/2305.17680/) (2023)
- [From Instructions to Intrinsic Human Values -- A Survey of Alignment Goals for Big Models](https://mlsanity.com/paper/2308.12014/) (2023)
- [Speaking Multiple Languages Affects the Moral Bias of Language Models](https://mlsanity.com/paper/2211.07733/) (2022)
- [Towards an Atlas of Cultural Commonsense for Machine Reasoning](https://mlsanity.com/paper/2009.05664/) (2020)
- [Unveiling Gender Bias in Terms of Profession Across LLMs: Analyzing and Addressing Sociological Implications](https://mlsanity.com/paper/2307.09162/) (2023)
- [What does ChatGPT return about human values? Exploring value bias in ChatGPT using a descriptive value theory](https://mlsanity.com/paper/2304.03612/) (2023)
- [Your spouse needs professional help: Determining the Contextual Appropriateness of Messages through Modeling Social Relationships](https://mlsanity.com/paper/2307.02763/) (2023)
- [Cultural Differences in Friendship Network Behaviors: A Snapchat Case Study](https://mlsanity.com/paper/2301.13801/) (2023)

## Built on by (most cited first, 24 of 24)

- [NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models](https://mlsanity.com/paper/2404.12464/) (2024)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [Beyond Aesthetics: Cultural Competence in Text-to-Image Models](https://mlsanity.com/paper/2407.06863/) (2024)
- [Extrinsic Evaluation of Cultural Competence in Large Language Models](https://mlsanity.com/paper/2406.11565/) (2024)
- [SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture](https://mlsanity.com/paper/2506.15355/) (2025)
- [DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culture](https://mlsanity.com/paper/2509.19274/) (2025)
- [The Future of Open Human Feedback](https://mlsanity.com/paper/2408.16961/) (2024)
- [TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories](https://mlsanity.com/paper/2511.21322/) (2025)
- [Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness](https://mlsanity.com/paper/2502.09637/) (2025)
- [DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context](https://mlsanity.com/paper/2509.17399/) (2025)
- [KAHANI: Culturally-Nuanced Visual Storytelling Tool for Non-Western Cultures](https://mlsanity.com/paper/2410.19419/) (2024)
- [Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino](https://mlsanity.com/paper/2409.15380/) (2024)
- [MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs](https://mlsanity.com/paper/2507.09701/) (2025)
- [Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports](https://mlsanity.com/paper/2510.01247/) (2025)
- [Common to Whom? Regional Cultural Commonsense and LLM Bias in India](https://mlsanity.com/paper/2601.15550/) (2026)
- [Communicate to Play: Pragmatic Reasoning for Efficient Cross-Cultural Communication in Codenames](https://mlsanity.com/paper/2408.04900/) (2024)
- [Culture Cartography: Mapping the Landscape of Cultural Knowledge](https://mlsanity.com/paper/2510.27672/) (2025)
- [ELR-1000: A Community-Generated Dataset for Endangered Indic Indigenous Languages](https://mlsanity.com/paper/2512.01077/) (2025)
- [A Unified Framework to Quantify Cultural Intelligence of AI](https://mlsanity.com/paper/2603.01211/) (2026)
- [BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources](https://mlsanity.com/paper/2604.18423/) (2026)
- [The Curious Case of Curiosity across Human Cultures and LLMs](https://mlsanity.com/paper/2510.12943/) (2025)
- [To Generate or Discriminate? Methodological Considerations for Measuring Cultural Alignment in LLMs](https://mlsanity.com/paper/2601.02858/) (2026)
- [Far Out: Evaluating Language Models on Slang in Australian and Indian English](https://mlsanity.com/paper/2602.15373/) (2026)
- [CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systems](https://mlsanity.com/paper/2609.22494/) (2026)

## Abstract (arXiv)

Generative models are increasingly being used in various applications, such as text generation, commonsense reasoning, and question-answering. To be effective globally, these models must be aware of and account for local socio-cultural contexts, making it necessary to have benchmarks to evaluate the models for their cultural familiarity. Since the training data for LLMs is web-based and the Web is limited in its representation of information, it does not capture knowledge present within communities that are not on the Web. Thus, these models exacerbate the inequities, semantic misalignment, and stereotypes from the Web. There has been a growing call for community-centered participatory research methods in NLP. In this work, we respond to this call by using participatory research methods to introduce textitDOSA, the first community-generated Dataset of 615 Social Artifacts, by engaging with 260 participants from 19 different Indian geographic subcultures. We use a gamified framework that relies on collective sensemaking to collect the names and descriptions of these artifacts such that the descriptions semantically align with the shared sensibilities of the individuals from those cultures. Next, we benchmark four popular LLMs and find that they show significant variation across regional sub-cultures in their ability to infer the artifacts.