mlsanityData to 14 Oct 2026

INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge

arXiv 2411.19799 ↗First posted 29 Nov 2024cs.CLcited by 86

Not on the map. Why?

Abstract (arXiv)

The performance differential of large language models (LLM) between languages hinders their effective deployment in many regions, inhibiting the potential economic and societal value of generative AI tools in many communities. However, the development of functional LLMs in many languages (ie, multilingual LLMs) is bottlenecked by the lack of high-quality evaluation resources in languages other than English. Moreover, current practices in multilingual benchmark construction often translate English resources, ignoring the regional and cultural knowledge of the environments in which multilingual systems would be used. In this work, we construct an evaluation suite of 197,243 QA pairs from local exam sources to measure the capabilities of multilingual LLMs in a variety of regional contexts. Our novel resource, INCLUDE, is a comprehensive knowledge- and reasoning-centric benchmark across 44 written languages that evaluates multilingual LLMs for performance in the actual language environments where they would be deployed.

Key facts. Cited by 86 papers in the map (data through 14 Oct 2026). Builds on 70 of its 80 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

70 of 80 references.

References · 70

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  12. 2023Mistral 7Bcited by 3,164
38 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  9. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  10. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  11. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  12. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  13. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  14. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  15. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  16. 2023Mathematical Capabilities of ChatGPTcited by 268
  17. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  18. 2020XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generationcited by 222
  19. 2024A Careful Examination of Large Language Model Performance on Grade School Arithmeticcited by 191
  20. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  21. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  22. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  23. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  24. 2023Cultural Bias and Cultural Alignment of Large Language Modelscited by 149
  25. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  26. 2024Investigating Cultural Alignment of Large Language Modelscited by 143
  27. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  28. 2021Gender Bias in Machine Translationcited by 117
  29. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 109
  30. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 103
  31. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  32. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  33. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabiccited by 88
  34. 2023Spanish Pre-trained BERT Model and Evaluation Datacited by 81
  35. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  36. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  37. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  38. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 29; 2026: 55.

86 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2025Qwen3-VL Technical Reportcited by 2,793
  3. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  4. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  5. 2026On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stabilitycited by 35
  6. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
  7. 2025BenchMAX: A Comprehensive Multilingual Evaluation Suite for Large Language Modelscited by 23
  8. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  9. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 20
  10. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  11. 2025MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languagescited by 15
  12. 2026ERNIE 5.0 Technical Reportcited by 11
38 more
  1. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  2. 2025The Multilingual Divide and Its Impact on Global AI Safetycited by 10
  3. 2025BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Datacited by 10
  4. 2025Revisiting Multilingual Data Mixtures in Language Model Pretrainingcited by 10
  5. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  6. 2025Hunyuan-MT Technical Reportcited by 7
  7. 2025HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Modelscited by 7
  8. 2026SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-trainingcited by 7
  9. 2025SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhalacited by 6
  10. 2025Bielik 11B v3: Multilingual Large Language Model for European Languagescited by 6
  11. 2026Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistencycited by 6
  12. 2025The Art of Asking: Multilingual Prompt Optimization for Synthetic Datacited by 5
  13. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  14. 2025CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Datasetcited by 4
  15. 2025UPDESH: Synthesizing Grounded Instruction Tuning Data for 13 Indic Languagescited by 4
  16. 2025Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Modelcited by 4
  17. 2025TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behaviorcited by 4
  18. 2026Bielik-Minitron-7B: Compressing Large Language Models via Structured Pruning and Knowledge Distillation for the Polish Languagecited by 4
  19. 2026Location Not Found: Exposing Implicit Local and Global Biases in Multilingual LLMscited by 4
  20. 2026MemSFT: Mitigating Alignment Tax with an External Parametric Memorycited by 3
  21. 2026Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Trainingcited by 3
  22. 2025Camellia: Benchmarking Cultural Biases in LLMs for Asian Languagescited by 2
  23. 2026Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmarkcited by 2
  24. 2026Round-Trip Translation Reveals What Frontier Multilingual Benchmarks Misscited by 2
  25. 2025Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Modelscited by 1
  26. 2025MEDAL: A Framework for Benchmarking LLMs as Multilingual Open-Domain Dialogue Evaluatorscited by 1
  27. 2025Test Set Quality in Multilingual LLM Evaluationcited by 1
  28. 2025BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culturecited by 1
  29. 2026No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understandingcited by 1
  30. 2026CulturALL: Benchmarking Multilingual and Multicultural Competence of LLMs on Grounded Taskscited by 1
  31. 2026The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Modelscited by 1
  32. 2026Defining Cultural Capabilities for AI Evaluation: A Taxonomy Grounded in Intercultural Communication Theorycited by 1
  33. 2026Learning to Route Languages for Multilingual Policy Optimizationcited by 1
  34. 2026CARTE: A Benchmark for Mapping Language Model Knowledge Across Francecited by 1
  35. 2026MADE: Beyond Scoring via a Multilingual Agentic Diagnosing Engine for Fine-Grained Evaluation Insightscited by 1
  36. 2026Structural priors for data-efficient language learningcited by 1
  37. 2026HakemBench: A Turkish Benchmark of Typed Decisionscited by 1
  38. 2025MATA: Mindful Assessment of the Telugu Abilities of Large Language Modelscited by 0

50 most cited of 86 All 86 (JSON)