mlsanityData to 7 Oct 2026

COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances

arXiv 2311.01012 ↗First posted 2 Nov 2023cs.CLcited by 35

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We present COPAL-ID, a novel, public Indonesian language common sense reasoning dataset. Unlike the previous Indonesian COPA dataset (XCOPA-ID), COPAL-ID incorporates Indonesian local and cultural nuances, and therefore, provides a more natural portrayal of day-to-day causal reasoning within the Indonesian cultural sphere. Professionally written by natives from scratch, COPAL-ID is more fluent and free from awkward phrases, unlike the translated XCOPA-ID. In addition, we present COPAL-ID in both standard Indonesian and in Jakartan Indonesian-a dialect commonly used in daily conversation. COPAL-ID poses a greater challenge for existing open-sourced and closed state-of-the-art multilingual language models, yet is trivially easy for humans. Our findings suggest that general multilingual models struggle to perform well, achieving 66.91% accuracy on COPAL-ID. South-East Asian-specific models achieve slightly better performance of 73.88% accuracy. Yet, this number still falls short of near-perfect human performance. This shows that these language models are still way behind in comprehending the local nuances of Indonesian.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 19 of its 35 references that are in the map. First posted on arXiv 2 Nov 2023.

Builds on

19 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU7 Oct
  2. 2023Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings15 Sep
  3. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performance23 May

Other references · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  5. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  6. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  7. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  8. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  9. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  10. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
  11. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 64
  12. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
4 more
  1. 2019How Language-Neutral is Multilingual BERT?cited by 51
  2. 2022Crosslingual Generalization through Multitask Finetuningcited by 26
  3. 2023BHASA: A Holistic Southeast Asian Linguistic and Cultural Evaluation Suite for Large Language Modelscited by 16
  4. 2020Semi-Supervised Low-Resource Style Transfer of Indonesian Informal to Formal Language with Iterative Forward-Translationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 14; 2026: 7.

35 papers.

Sort:

By area:

  1. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmarkcited by 118
  2. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  3. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  4. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  5. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 38
  6. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  7. 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledgecited by 28
  8. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  9. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  10. 2024Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaborationcited by 19
  11. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  12. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 17
23 more
  1. 2025Multilingual Language Model Pretraining using Machine-translated Datacited by 17
  2. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  3. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  4. 2025SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabiacited by 8
  5. 2024Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarioscited by 7
  6. 2025IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languagescited by 7
  7. 2024LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classificationcited by 6
  8. 2025LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languagescited by 5
  9. 2024Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesiacited by 4
  10. 2024Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipinocited by 3
  11. 2025MCEval: A Dynamic Framework for Fair Multilingual Cultural Evaluation of LLMscited by 3
  12. 2025Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMscited by 2
  13. 2025RusCode: Russian Cultural Code Benchmark for Text-to-Image Generationcited by 1
  14. 2025Social Bias in Multilingual Language Models: A Surveycited by 1
  15. 2026No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understandingcited by 1
  16. 2025CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generationcited by 0
  17. 2025'Too much alignment; not enough culture': Re-balancing cultural alignment practices in LLMscited by 0
  18. 2026To Generate or Discriminate? Methodological Considerations for Measuring Cultural Alignment in LLMscited by 0
  19. 2026Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Gamecited by 0
  20. 2026Injecting Knowledge from Social Science Journals to Improve Indonesian Cultural Understanding by LLMscited by 0
  21. 2026CultureTalk-ID: A Multi-Task Dialogue Benchmark for Cultural Commonsense in Indonesian Local Languagescited by 0
  22. 2026CultureMINE: Datasets and Methods for Improving the Cultural Capabilities of NLP Systemscited by 0
  23. 2026FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
  2. 2025Culturally-Nuanced Story Generation for Reasoning in Low-Resource Languages: The Case of Javanese and Sundanese
  3. 2026XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad
  4. 2025A Survey of Multilingual Reasoning in Language Models
  5. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs
  6. 2025A Survey on Enhancing Causal Reasoning Ability of Large Language Models
  7. 2025IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesian
  8. 2025BengaliFig: A Low-Resource Challenge for Figurative and Culturally Grounded Reasoning in Bengali