mlsanityData to 14 Oct 2026

Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU

arXiv 2310.04928 ↗First posted 7 Oct 2023cs.CLcited by 53

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Although large language models (LLMs) are often pre-trained on large-scale multilingual texts, their reasoning abilities and real-world knowledge are mainly evaluated based on English datasets. Assessing LLM capabilities beyond English is increasingly vital but hindered due to the lack of suitable datasets. In this work, we introduce IndoMMLU, the first multi-task language understanding benchmark for Indonesian culture and languages, which consists of questions from primary school to university entrance exams in Indonesia. By employing professional teachers, we obtain 14,981 questions across 64 tasks and education levels, with 46% of the questions focusing on assessing proficiency in the Indonesian language and knowledge of nine local languages and cultures in Indonesia. Our empirical evaluations show that GPT-3.5 only manages to pass the Indonesian primary school level, with limited knowledge of local Indonesian languages and culture. Other smaller models such as BLOOMZ and Falcon perform at even lower levels.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 41 of its 61 references that are in the map. First posted on arXiv 7 Oct 2023.

Builds on

41 of 61 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 40

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
28 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  3. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  5. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  7. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  8. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 285
  9. 2022Prompting GPT-3 To Be Reliablecited by 260
  10. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  11. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  12. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  13. 2022NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Taskscited by 112
  14. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
  15. 2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 81
  16. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  17. 2020IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLPcited by 58
  18. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  19. 2020A Survey on Machine Reading Comprehension Systemscited by 41
  20. 2022PERFECT: Prompt-free and Efficient Few-shot Learning with Language Modelscited by 40
  21. 2023ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmarkcited by 33
  22. 2023M3KE: A Massive Multi-Level Multi-Subject Knowledge Evaluation Benchmark for Chinese Large Language Modelscited by 30
  23. 2021IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initializationcited by 25
  24. 2020Improving Bi-LSTM Performance for Indonesian Sentiment Analysis Using Paragraph Vectorcited by 18
  25. 2020Towards Computational Linguistics in Minangkabau Language: Studies on Sentiment Analysis and Machine Translationcited by 16
  26. 2020Liputan6: A Large-scale Indonesian Dataset for Text Summarizationcited by 14
  27. 2022Improving negation detection with negation-focused pre-trainingcited by 14
  28. 2022IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehensioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 20; 2026: 11.

53 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  3. 2024KMMLU: Measuring Massive Multitask Language Understanding in Koreancited by 91
  4. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 88
  5. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  6. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  7. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Modelscited by 41
  8. 2024IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provincescited by 39
  9. 2024Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanesecited by 36
  10. 2023COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuancescited by 35
  11. 2024CaLMQA: Exploring culturally specific long-form question answering across 23 languagescited by 31
  12. 2025The Bitter Lesson Learned from 2,000+ Multilingual Benchmarkscited by 31
38 more
  1. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  2. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
  3. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  4. 2024Cultural Conditioning or Placebo? On the Effectiveness of Socio-Demographic Promptingcited by 22
  5. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  6. 2024Cendol: Open Instruction-tuned Generative Large Language Models for Indonesian Languagescited by 18
  7. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  8. 2024Komodo: A Linguistic Expedition into Indonesia's Regional Languagescited by 10
  9. 2025Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Abilitycited by 10
  10. 2025Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asiacited by 9
  11. 2025Multilingual Performance Biases of Large Language Models in Educationcited by 9
  12. 2025MultiHoax: A Dataset of Multi-hop False-Premise Questionscited by 9
  13. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  14. 2024Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Modelscited by 7
  15. 2025IndoSafety: Culturally Grounded Safety for LLMs in Indonesian Languagescited by 7
  16. 2025KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstancited by 6
  17. 2025LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languagescited by 6
  18. 2025SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhalacited by 6
  19. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  20. 2025Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptationcited by 5
  21. 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 5
  22. 2024BEnQA: A Question Answering and Reasoning Benchmark for Bengali and Englishcited by 4
  23. 2024Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesiacited by 4
  24. 2024MERaLiON-TextLLM: Cross-Lingual Understanding of Large Language Models in Chinese, Indonesian, Malay, and Singlishcited by 4
  25. 2025Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Modelscited by 4
  26. 2025Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answeringcited by 3
  27. 2025What Do Indonesians Really Need from Language Technology? A Nationwide Surveycited by 3
  28. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  29. 2024Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Modelscited by 2
  30. 2025LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriamacited by 2
  31. 2025Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgishcited by 2
  32. 2025A Qualitative Investigation into LLM-Generated Multilingual Code Comments and Automatic Evaluation Metricscited by 2
  33. 2026EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Examscited by 1
  34. 2026Positional Cognitive Specialization: Where Do LLMs Learn To Comprehend and Speak Your Language?cited by 1
  35. 2026Injecting Knowledge from Social Science Journals to Improve Indonesian Cultural Understanding by LLMscited by 0
  36. 2026Toward LLMs Beyond English-Centric Developmentcited by 0
  37. 2026DEPART: DEcomposing PARiTy across Multilingual LLMscited by 0
  38. 2026UrduMMLU: A Massive Multitask Benchmark for Urdu Language Understandingcited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  2. 2024Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT
  3. 2024TurkishMMLU: Measuring Massive Multitask Language Understanding in Turkish
  4. 2025MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language
  5. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models
  6. 2024A Survey of Large Language Models for European Languages
  7. 2025Measuring Hong Kong Massive Multi-Task Language Understanding
  8. 2024Compass: Large Multilingual Language Model for South-east Asia