# Y-NQ: English-Yor`ub'a Evaluation dataset for Open-Book Reading Comprehension and Text Generation

arXiv: https://arxiv.org/abs/2412.08279 · HTML: https://mlsanity.com/paper/2412.08279/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 10 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Builds on (most cited first, 6 of 6)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialects](https://mlsanity.com/paper/2406.19564/) (2024)
- [NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages](https://mlsanity.com/paper/2308.09768/) (2023)

## Built on by (most cited first, 1 of 1)

- [Can AI Validate Science? Benchmarking LLMs for Accurate Scientific Claim rightarrow Evidence Reasoning](https://mlsanity.com/paper/2506.08235/) (2025)

## Abstract (arXiv)

The purpose of this work is to share an English-Yor`ub'a evaluation dataset for open-book reading comprehension and text generation to assess the performance of models both in a high- and a low- resource language. The dataset contains 358 questions and answers on 338 English documents and 208 Yor`ub'a documents. The average document length is ~ 10k words for English and 430 words for Yor`ub'a. Experiments show a consistent disparity in performance between the two languages, with Yor`ub'a falling behind English for automatic metrics even if documents are much shorter for this language. For a small set of documents with comparable length, performance of Yor`ub'a drops by x2.5 times. When analyzing performance by length, we observe that Yor`ub'a decreases performance dramatically for documents that reach 1500 words while English performance is barely affected at that length. Our dataset opens the door to showcasing if English LLM reading comprehension capabilities extend to Yor`ub'a, which for the evaluated LLMs is not the case.