mlsanityData to 14 Oct 2026

MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages

arXiv 2504.10356 ↗First posted 14 Apr 2025cs.CLcited by 26

Not on the map. Why?

Abstract (arXiv)

We present MultiLoKo, a new benchmark for evaluating multilinguality in LLMs covering 31 languages. MultiLoKo consists of three partitions: a main partition consisting of 500 questions per language, separately sourced to be locally relevant to the specific language, and two translated partitions, containing human-authored translations from 30 non-English languages to English and vice versa. For comparison, we also release corresponding machine-authored translations. The data is equally distributed over two splits: a dev split and a blind, out-of-distribution test split. MultiLoKo can be used to study a variety of questions regarding the multilinguality of LLMs as well as meta-questions about multilingual benchmark creation. We compute MultiLoKo scores for 11 base and chat models marketed to be multilingual and study their average performance, their performance parity across languages, how much their ability to answer questions depends on the question language, and which languages are most difficult. None of the models we studied performs well on MultiLoKo, as indicated by low average scores as well as large differences between the best and worst scoring languages. Furthermore, we find a substantial effect of the question language, indicating sub-optimal knowledge transfer between languages. Lastly, we find that using local vs English-translated data can result in differences more than 20 points for the best performing models, drastically change the estimated difficulty of some languages. For using machines instead of human translations, we find a weaker effect on ordering of language difficulty, a larger difference in model rankings, and a substantial drop in estimated performance for all models.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 36 of its 44 references that are in the map. First posted on arXiv 14 Apr 2025.

Builds on

36 of 44 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  4. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  6. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  7. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  8. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  9. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  10. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  11. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  12. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
24 more
  1. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  2. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  3. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  4. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  5. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  6. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  7. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  8. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  9. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  10. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  11. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 86
  12. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluationcited by 79
  13. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 72
  14. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  15. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  16. 2024Quantifying Variance in Evaluation Benchmarkscited by 54
  17. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLUcited by 53
  18. 2023Mind the instructions: a holistic evaluation of consistency and interactions in prompt-based learningcited by 31
  19. 2024MERA: A Comprehensive LLM Evaluation in Russiancited by 22
  20. 2024Linguini: A benchmark for language-agnostic linguistic reasoningcited by 17
  21. 2024Dallah: A Dialect-Aware Multimodal Large Language Model for Arabiccited by 16
  22. 2023Separating form and meaning: Using self-consistency to quantify task understanding across multiple sensescited by 15
  23. 2024From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistencycited by 7
  24. 2024Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models?cited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 15.

26 papers.

Sort:

By area:

  1. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  2. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMscited by 16
  3. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 15
  4. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
  5. 2025Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Culturescited by 10
  6. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  7. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  8. 2025MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretrainingcited by 5
  9. 2025CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Datasetcited by 4
  10. 2025Rethinking Cross-lingual Gaps from a Statistical Viewpointcited by 4
  11. 2025MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languagescited by 3
  12. 2026Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scriptscited by 3
14 more
  1. 2025CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Modelscited by 2
  2. 2025Test Set Quality in Multilingual LLM Evaluationcited by 1
  3. 2025Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domainscited by 1
  4. 2026M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Modelscited by 1
  5. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generationscited by 0
  6. 2026When English Rewrites Local Knowledge: Global Narrative Dominance in Large Language Modelscited by 0
  7. 2026MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmarkcited by 0
  8. 2026Language Equality has a Price: A Systematic Investigation of Multi-turn LLM Performance for EU-24+cited by 0
  9. 2026EuroAlpaca: Task-Preserving Localisation of Instruction Data for European Languagescited by 0
  10. 2026Do LLMs Make More Mistakes If They Do Not Believe the Input Data?cited by 0
  11. 2026Large-scale factor analysis shows machine intelligence is only partially interpretablecited by 0
  12. 2026Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparitiescited by 0
  13. 2026Cross-Lingual Alignment for Decoder-Only Models using MoE Routerscited by 0
  14. 2026sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovakcited by 0