# COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances

arXiv: https://arxiv.org/abs/2311.01012 · HTML: https://mlsanity.com/paper/2311.01012/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 2 Nov 2023.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories](https://mlsanity.com/paper/1604.01696/) (2016)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Few-shot Learning with Multilingual Language Models](https://mlsanity.com/paper/2112.10668/) (2021)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning](https://mlsanity.com/paper/2106.06937/) (2021)
- [PolyLM: An Open Source Polyglot Large Language Model](https://mlsanity.com/paper/2307.06018/) (2023)
- [Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU](https://mlsanity.com/paper/2310.04928/) (2023)
- [How Language-Neutral is Multilingual BERT?](https://mlsanity.com/paper/1911.03310/) (2019)
- [LLM-powered Data Augmentation for Enhanced Cross-lingual Performance](https://mlsanity.com/paper/2305.14288/) (2023)
- [Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings](https://mlsanity.com/paper/2309.08591/) (2023)
- [Crosslingual Generalization through Multitask Finetuning](https://mlsanity.com/paper/2211.01786/) (2022)
- [BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Models](https://mlsanity.com/paper/2309.06085/) (2023)
- [Semi-Supervised Low-Resource Style Transfer of Indonesian Informal to Formal Language with Iterative Forward-Translation](https://mlsanity.com/paper/2011.03286/) (2020)

## Built on by (most cited first, 30 of 35)

- [CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark](https://mlsanity.com/paper/2406.05967/) (2024)
- [CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Prompting](https://mlsanity.com/paper/2404.10199/) (2024)
- [Crosslingual Reasoning through Test-Time Scaling](https://mlsanity.com/paper/2505.05408/) (2025)
- [SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://mlsanity.com/paper/2406.10118/) (2024)
- [IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces](https://mlsanity.com/paper/2404.01854/) (2024)
- [CaLMQA: Exploring culturally specific long-form question answering across 23 languages](https://mlsanity.com/paper/2406.17761/) (2024)
- [CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge](https://mlsanity.com/paper/2404.06664/) (2024)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [BertaQA: How Much Do Language Models Know About Local Culture?](https://mlsanity.com/paper/2406.07302/) (2024)
- [Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration](https://mlsanity.com/paper/2406.16469/) (2024)
- [Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languages](https://mlsanity.com/paper/2404.06138/) (2024)
- [Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs](https://mlsanity.com/paper/2502.12982/) (2025)
- [Multilingual Language Model Pretraining using Machine-translated Data](https://mlsanity.com/paper/2502.13252/) (2025)
- [Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models](https://mlsanity.com/paper/2502.14906/) (2025)
- [Meta-Cultural Competence: Climbing the Right Hill of Cultural Awareness](https://mlsanity.com/paper/2502.09637/) (2025)
- [SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia](https://mlsanity.com/paper/2503.17485/) (2025)
- [Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios](https://mlsanity.com/paper/2406.00343/) (2024)
- [IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languages](https://mlsanity.com/paper/2506.02573/) (2025)
- [LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classification](https://mlsanity.com/paper/2411.19638/) (2024)
- [LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languages](https://mlsanity.com/paper/2508.12459/) (2025)
- [Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia](https://mlsanity.com/paper/2409.08564/) (2024)
- [Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino](https://mlsanity.com/paper/2409.15380/) (2024)
- [MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMs](https://mlsanity.com/paper/2507.09701/) (2025)
- [Social Bias in Multilingual Language Models: A Survey](https://mlsanity.com/paper/2508.20201/) (2025)
- [Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMs](https://mlsanity.com/paper/2510.14565/) (2025)
- [RusCode: Russian Cultural Code Benchmark for Text-to-Image Generation](https://mlsanity.com/paper/2502.07455/) (2025)
- [No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding](https://mlsanity.com/paper/2602.03709/) (2026)
- [CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation](https://mlsanity.com/paper/2505.04851/) (2025)
- ['Too much alignment; not enough culture': Re-balancing cultural alignment practices in LLMs](https://mlsanity.com/paper/2509.26167/) (2025)
- [To Generate or Discriminate? Methodological Considerations for Measuring Cultural Alignment in LLMs](https://mlsanity.com/paper/2601.02858/) (2026)

## Abstract (arXiv)

We present COPAL-ID, a novel, public Indonesian language common sense reasoning dataset. Unlike the previous Indonesian COPA dataset (XCOPA-ID), COPAL-ID incorporates Indonesian local and cultural nuances, and therefore, provides a more natural portrayal of day-to-day causal reasoning within the Indonesian cultural sphere. Professionally written by natives from scratch, COPAL-ID is more fluent and free from awkward phrases, unlike the translated XCOPA-ID. In addition, we present COPAL-ID in both standard Indonesian and in Jakartan Indonesian-a dialect commonly used in daily conversation. COPAL-ID poses a greater challenge for existing open-sourced and closed state-of-the-art multilingual language models, yet is trivially easy for humans. Our findings suggest that general multilingual models struggle to perform well, achieving 66.91% accuracy on COPAL-ID. South-East Asian-specific models achieve slightly better performance of 73.88% accuracy. Yet, this number still falls short of near-perfect human performance. This shows that these language models are still way behind in comprehending the local nuances of Indonesian.