# MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages

arXiv: https://arxiv.org/abs/2509.04111 · HTML: https://mlsanity.com/paper/2509.04111/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 4 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Adversarial Examples for Evaluating Reading Comprehension Systems](https://mlsanity.com/paper/1707.07328/) (2017)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering](https://mlsanity.com/paper/2007.15207/) (2020)
- [FQuAD: French Question Answering Dataset](https://mlsanity.com/paper/2002.06071/) (2020)
- [SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis](https://mlsanity.com/paper/1912.09723/) (2019)
- [GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval](https://mlsanity.com/paper/2104.12741/) (2021)
- [Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks](https://mlsanity.com/paper/2406.13469/) (2024)
- [Making Neural QA as Simple as Possible but not Simpler](https://mlsanity.com/paper/1703.04816/) (2017)
- [NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian](https://mlsanity.com/paper/2312.01314/) (2023)
- [FoQA: A Faroese Question-Answering Dataset](https://mlsanity.com/paper/2502.07642/) (2025)
- [Enhancing Portuguese Variety Identification with Cross-Domain Approaches](https://mlsanity.com/paper/2502.14394/) (2025)
- [Korean-Specific Dataset for Table Question Answering](https://mlsanity.com/paper/2201.06223/) (2022)

## Built on by (most cited first, 9 of 9)

- [A Diagnostic Benchmark for Sweden-Related Factual Knowledge](https://mlsanity.com/paper/2510.21360/) (2025)
- [Family Matters: Language Transfer and Merging for Adapting Small LLMs to Faroese](https://mlsanity.com/paper/2510.00810/) (2025)
- [BYOL: Bring Your Own Language Into LLMs](https://mlsanity.com/paper/2601.10804/) (2026)
- [SDUs DAISY: A Benchmark for Danish Culture](https://mlsanity.com/paper/2601.19930/) (2026)
- [Pretraining and Benchmarking Modern Encoders for Latvian](https://mlsanity.com/paper/2603.15005/) (2026)
- [Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic](https://mlsanity.com/paper/2603.16406/) (2026)
- [A multilingual hallucination benchmark: MultiWikiQHalluA](https://mlsanity.com/paper/2605.02504/) (2026)
- [CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations](https://mlsanity.com/paper/2605.26293/) (2026)
- [sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak](https://mlsanity.com/paper/2610.09152/) (2026)

## Abstract (arXiv)

We introduce a new reading comprehension dataset, dubbed MultiWikiQA, which covers 306 languages and has 1,220,757 samples in total. We start with Wikipedia articles, which also provide the context for the dataset samples, and use an LLM to generate question/answer pairs related to the Wikipedia article, ensuring that the answer appears verbatim within the article. Next, the question is then rephrased to hinder simple word matching methods from performing well on the dataset. We conduct a crowdsourced human evaluation of the fluency of the generated questions, which included 156 respondents across 30 of the languages (both low- and high-resource). All 30 languages received a mean fluency rating above ``mostly natural'', showing that the samples are of good quality. We evaluate 6 different language models, both decoder and encoder models of varying sizes, showing that the benchmark is sufficiently difficult and that there is a large performance discrepancy amongst the languages. Both the dataset and survey evaluations are publicly available.