# Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

arXiv: https://arxiv.org/abs/2402.17302 · HTML: https://mlsanity.com/paper/2402.17302/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 51 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Builds on (most cited first, 17 of 17)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [On the Risk of Misinformation Pollution with Large Language Models](https://mlsanity.com/paper/2305.13661/) (2023)
- [NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages](https://mlsanity.com/paper/2205.15960/) (2022)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU](https://mlsanity.com/paper/2310.04928/) (2023)
- [MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge](https://mlsanity.com/paper/1803.05223/) (2018)
- [Large Malaysian Language Model Based on Mistral for Enhanced Local Language Understanding](https://mlsanity.com/paper/2401.13565/) (2024)

## Built on by (most cited first, 30 of 36)

- [BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages](https://mlsanity.com/paper/2406.09948/) (2024)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces](https://mlsanity.com/paper/2404.01854/) (2024)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [CulFiT: A Fine-grained Cultural-aware LLM Training Paradigm via Multilingual Critique Data Synthesis](https://mlsanity.com/paper/2505.19484/) (2025)
- [Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs](https://mlsanity.com/paper/2510.16062/) (2025)
- [A Survey on Human-Centric LLMs](https://mlsanity.com/paper/2411.14491/) (2024)
- [mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humans](https://mlsanity.com/paper/2406.04215/) (2024)
- [Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness](https://mlsanity.com/paper/2502.09637/) (2025)
- [SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia](https://mlsanity.com/paper/2503.17485/) (2025)
- [Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models](https://mlsanity.com/paper/2409.01584/) (2024)
- [ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research](https://mlsanity.com/paper/2506.11117/) (2025)
- [Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability](https://mlsanity.com/paper/2506.01789/) (2025)
- [CultureSynth: A Hierarchical Taxonomy-Guided and Retrieval-Augmented Framework for Cultural Question-Answer Synthesis](https://mlsanity.com/paper/2509.10886/) (2025)
- [Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreation](https://mlsanity.com/paper/2506.01565/) (2025)
- [What are Foundation Models Cooking in the Post-Soviet World?](https://mlsanity.com/paper/2502.18583/) (2025)
- [Can LLMs Grasp Implicit Cultural Values? Benchmarking LLMs' Cultural Intelligence with CQ-Bench](https://mlsanity.com/paper/2504.01127/) (2025)
- [Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering](https://mlsanity.com/paper/2505.16591/) (2025)
- [Synthetic Voice Data for Automatic Speech Recognition in African Languages](https://mlsanity.com/paper/2507.17578/) (2025)
- [Preserving Cultural Identity with Context-Aware Translation Through Multi-Agent AI Systems](https://mlsanity.com/paper/2503.04827/) (2025)
- [The Unintended Trade-off of AI Alignment:Balancing Hallucination Mitigation and Safety in LLMs](https://mlsanity.com/paper/2510.07775/) (2025)
- [CALM: Culturally Self-Aware Language Models](https://mlsanity.com/paper/2601.03483/) (2026)
- [Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification](https://mlsanity.com/paper/2601.16278/) (2026)
- [Finding A Voice: Exploring the Potential of African American Dialect and Voice Generation for Chatbots](https://mlsanity.com/paper/2501.03441/) (2025)
- [RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets](https://mlsanity.com/paper/2510.06143/) (2025)
- [No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding](https://mlsanity.com/paper/2602.03709/) (2026)
- [Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments](https://mlsanity.com/paper/2412.12417/) (2024)
- [NativQA Framework: Enabling LLMs and VLMs with Native, Local, and Everyday Knowledge](https://mlsanity.com/paper/2504.05995/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly being used to generate synthetic data for training and evaluating models. However, it is unclear whether they can generate a good quality of question answering (QA) dataset that incorporates knowledge and cultural nuance embedded in a language, especially for low-resource languages. In this study, we investigate the effectiveness of using LLMs in generating culturally relevant commonsense QA datasets for Indonesian and Sundanese languages. To do so, we create datasets for these languages using various methods involving both LLMs and human annotators, resulting in ~4.5K questions per language (~9K in total), making our dataset the largest of its kind. Our experiments show that automatic data adaptation from an existing English dataset is less effective for Sundanese. Interestingly, using the direct generation method on the target language, GPT-4 Turbo can generate questions with adequate general knowledge in both languages, albeit not as culturally 'deep' as humans. We also observe a higher occurrence of fluency errors in the Sundanese dataset, highlighting the discrepancy between medium- and lower-resource languages.