mlsanityData to 14 Oct 2026

The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants

arXiv 2308.16884 ↗First posted 31 Aug 2023cs.CL cs.AI cs.LGcited by 311

Not on the map. Why?

Abstract (arXiv)

We present Belebele, a multiple-choice machine reading comprehension (MRC) dataset spanning 122 language variants. Significantly expanding the language coverage of natural language understanding (NLU) benchmarks, this dataset enables the evaluation of text models in high-, medium-, and low-resource languages. Each question is based on a short passage from the Flores-200 dataset and has four multiple-choice answers. The questions were carefully curated to discriminate between models with different levels of general language comprehension. The English dataset on its own proves difficult enough to challenge state-of-the-art language models. Being fully parallel, this dataset enables direct comparison of model performance across all languages. We use this dataset to evaluate the capabilities of multilingual masked language models (MLMs) and large language models (LLMs). We present extensive results and find that despite significant cross-lingual transfer in English-centric LLMs, much smaller MLMs pretrained on balanced multilingual data still understand far more languages. We also observe that larger vocabulary size and conscious vocabulary construction correlate with better performance on low-resource languages. Overall, Belebele opens up new avenues for evaluating and analyzing the multilingual capabilities of NLP systems.

Key facts. Cited by 311 papers in the map (data through 14 Oct 2026). Builds on 62 of its 91 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

62 of 91 references.

References · 62

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
38 more
  1. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  3. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  4. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  5. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  6. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  7. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  8. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  11. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  12. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  13. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  14. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  15. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  16. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  17. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  18. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  19. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  20. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  21. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  22. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  23. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  24. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  25. 2016Transfer Learning for Low-Resource Neural Machine Translationcited by 188
  26. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  27. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  28. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  29. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  30. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  31. 2022Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answeringcited by 72
  32. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 65
  33. 2021Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoningcited by 65
  34. 2020EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answeringcited by 60
  35. 2020FQuAD: French Question Answering Datasetcited by 58
  36. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  37. 2023Large Language Models as Superpositions of Cultural Perspectivescited by 49
  38. 2019Neural Arabic Question Answeringcited by 48

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 89; 2025: 101; 2026: 113.

311 papers.

Sort:

By area:

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  5. 2024LLMs Are Few-Shot In-Context Low-Resource Language Learnerscited by 68
  6. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Modelscited by 67
  7. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  8. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  9. 2024Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluationcited by 57
  10. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  11. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 53
  12. 2024DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languagescited by 50
38 more
  1. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  2. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  3. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  4. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  5. 2024Latxa: An Open Language Model and Evaluation Suite for Basquecited by 42
  6. 2024Apollo: A Lightweight Multilingual Medical LLM towards Democratizing Medical AI to 6B Peoplecited by 38
  7. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  8. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  9. 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selectioncited by 35
  10. 2024Code-Switching Red-Teaming: LLM Evaluation for Safety and Multilingual Understandingcited by 34
  11. 2026When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturationcited by 33
  12. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32
  13. 2024Multi-Task Inference: Can Large Language Models Follow Multiple Instructions at Once?cited by 30
  14. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMscited by 30
  15. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  16. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 29
  17. 2025Multilingual Routing in Mixture-of-Expertscited by 29
  18. 2024Evaluating the Elementary Multilingual Capabilities of Large Language Models with MultiQcited by 27
  19. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
  20. 2024MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignmentcited by 25
  21. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  22. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  23. 2024EXAONE 3.0 7.8B Instruction Tuned Language Modelcited by 21
  24. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 21
  25. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 21
  26. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  27. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  28. 2024Sailor: Open Language Models for South-East Asiacited by 19
  29. 2024Unlocking the Potential of Model Merging for Low-Resource Languagescited by 19
  30. 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 19
  31. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
  32. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  33. 2025Plutus: Benchmarking Large Language Models in Low-Resource Greek Financecited by 18
  34. 2024Meltemi: The first open Large Language Model for Greekcited by 17
  35. 2024Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Expertscited by 17
  36. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  37. 2025Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulationcited by 17
  38. 2024MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routingcited by 16

50 most cited of 311 All 311 (JSON)