# Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languages

arXiv: https://arxiv.org/abs/2412.00948 · HTML: https://mlsanity.com/paper/2412.00948/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 1 Dec 2024.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI](https://mlsanity.com/paper/2310.16787/) (2023)
- [IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages](https://mlsanity.com/paper/2404.16816/) (2024)
- [LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languages](https://mlsanity.com/paper/2407.05975/) (2024)
- [IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models](https://mlsanity.com/paper/2406.03368/) (2024)
- [EtiCor: Corpus for Analyzing LLMs for Etiquettes](https://mlsanity.com/paper/2310.18974/) (2023)
- [AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages](https://mlsanity.com/paper/2305.06897/) (2023)
- [Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languages](https://mlsanity.com/paper/2305.08487/) (2023)
- [The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis](https://mlsanity.com/paper/2402.12976/) (2024)

## Built on by (most cited first, 8 of 8)

- [Multilingual Language Model Pretraining using Machine-translated Data](https://mlsanity.com/paper/2502.13252/) (2025)
- [AFRIDOC-MT: Document-level MT Corpus for African Languages](https://mlsanity.com/paper/2501.06374/) (2025)
- [TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages](https://mlsanity.com/paper/2502.11020/) (2025)
- [The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks](https://mlsanity.com/paper/2507.08538/) (2025)
- [Amplify Initiative: Building A Localized Data Platform for Globalized AI](https://mlsanity.com/paper/2504.14105/) (2025)
- [AI4Math: A Native Spanish Benchmark for University-Level Mathematical Reasoning in Large Language Models](https://mlsanity.com/paper/2505.18978/) (2025)
- [KatotohananQA: Evaluating Truthfulness of Large Language Models in Filipino](https://mlsanity.com/paper/2509.06065/) (2025)
- [Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation](https://mlsanity.com/paper/2610.20247/) (2026)

## Abstract (arXiv)

Evaluations of Large Language Models (LLMs) on knowledge-intensive tasks and factual accuracy often focus on high-resource languages primarily because datasets for low-resource languages (LRLs) are scarce. In this paper, we present Uhura -- a new benchmark that focuses on two tasks in six typologically-diverse African languages, created via human translation of existing English benchmarks. The first dataset, Uhura-ARC-Easy, is composed of multiple-choice science questions. The second, Uhura-TruthfulQA, is a safety benchmark testing the truthfulness of models on topics including health, law, finance, and politics. We highlight the challenges creating benchmarks with highly technical content for LRLs and outline mitigation strategies. Our evaluation reveals a significant performance gap between proprietary models such as GPT-4o and o1-preview, and Claude models, and open-source models like Meta's LLaMA and Google's Gemma. Additionally, all models perform better in English than in African languages. These results indicate that LMs struggle with answering scientific questions and are more prone to generating false claims in low-resource African languages. Our findings underscore the necessity for continuous improvement of multilingual LM capabilities in LRL settings to ensure safe and reliable use in real-world contexts. We open-source the Uhura Benchmark and Uhura Platform to foster further research and development in NLP for LRLs.