# AfroBench: How Good are Large Language Models on African Languages?

arXiv: https://arxiv.org/abs/2311.07978 · HTML: https://mlsanity.com/paper/2311.07978/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 102 references that are in the map.
- First posted on arXiv 14 Nov 2023.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Language Models are Multilingual Chain-of-Thought Reasoners](https://mlsanity.com/paper/2210.03057/) (2022)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages](https://mlsanity.com/paper/2302.08956/) (2023)
- [MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://mlsanity.com/paper/2311.07463/) (2023)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages](https://mlsanity.com/paper/2305.06897/) (2023)
- [MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages](https://mlsanity.com/paper/2305.13989/) (2023)
- [NollySenti: Leveraging Transfer Learning and Machine Translation for Nigerian Movie Sentiment Classification](https://mlsanity.com/paper/2305.10971/) (2023)
- [NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages](https://mlsanity.com/paper/2308.09768/) (2023)

## Built on by (most cited first, 30 of 43)

- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [Aya 23: Open Weight Releases to Further Multilingual Progress](https://mlsanity.com/paper/2405.15032/) (2024)
- [IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models](https://mlsanity.com/paper/2406.03368/) (2024)
- [How Does Quantization Affect Multilingual LLMs?](https://mlsanity.com/paper/2407.03211/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis](https://mlsanity.com/paper/2402.12976/) (2024)
- [One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers](https://mlsanity.com/paper/2506.10766/) (2025)
- [AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages](https://mlsanity.com/paper/2601.06395/) (2026)
- [Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?](https://mlsanity.com/paper/2403.01929/) (2024)
- [BabyBabelLM: A Multilingual Benchmark of Developmentally Plausible Training Data](https://mlsanity.com/paper/2510.10159/) (2025)
- [Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages](https://mlsanity.com/paper/2406.12739/) (2024)
- [mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks](https://mlsanity.com/paper/2506.08400/) (2025)
- [The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks](https://mlsanity.com/paper/2507.08538/) (2025)
- [A Three-Pronged Approach to Cross-Lingual Adaptation with Multilingual LLMs](https://mlsanity.com/paper/2406.17377/) (2024)
- [A Systematic Review of NLP for Ghanaian Languages: Datasets, Models, and a Research Roadmap](https://mlsanity.com/paper/2405.06818/) (2024)
- [Comparing LLM prompting with Cross-lingual transfer performance on Indigenous and Low-resource Brazilian Languages](https://mlsanity.com/paper/2404.18286/) (2024)
- [AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text](https://mlsanity.com/paper/2503.18247/) (2025)
- [AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages](https://mlsanity.com/paper/2510.23896/) (2025)
- [MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoning](https://mlsanity.com/paper/2509.08105/) (2025)
- [Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects](https://mlsanity.com/paper/2510.07890/) (2025)
- [Afri-MCQA: Multimodal Cultural Question Answering for African Languages](https://mlsanity.com/paper/2601.05699/) (2026)
- [AfriNLLB: Efficient Translation Models for African Languages](https://mlsanity.com/paper/2602.09373/) (2026)
- [Machine Intelligence in Africa: a survey](https://mlsanity.com/paper/2402.02218/) (2024)
- [Amplify Initiative: Building A Localized Data Platform for Globalized AI](https://mlsanity.com/paper/2504.14105/) (2025)
- [Evaluating LLMs for Career Guidance: Comparative Analysis of Computing Competency Recommendations Across Ten African Countries](https://mlsanity.com/paper/2510.18902/) (2025)
- [LLM-Powered Automatic Translation and Urgency in Crisis Scenarios](https://mlsanity.com/paper/2602.13452/) (2026)
- [SAFARI: A Community-Engaged Approach and Dataset of Stereotype Resources in the Sub-Saharan African Context](https://mlsanity.com/paper/2602.22404/) (2026)
- [AFRILANGTUTOR: Advancing Language Tutoring and Culture Education in Low-Resource Languages with Large Language Models](https://mlsanity.com/paper/2604.20996/) (2026)
- [Multilinguality at the Edge: Developing Language Models for the Global South](https://mlsanity.com/paper/2604.21637/) (2026)
- [Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languages](https://mlsanity.com/paper/2607.02235/) (2026)

## Abstract (arXiv)

Large-scale multilingual evaluations, such as MEGA, often include only a handful of African languages due to the scarcity of high-quality evaluation data and the limited discoverability of existing African datasets. This lack of representation hinders comprehensive LLM evaluation across a diverse range of languages and tasks. To address these challenges, we introduce AfroBench -- a multi-task benchmark for evaluating the performance of LLMs across 64 African languages, 15 tasks and 22 datasets. AfroBench consists of nine natural language understanding datasets, six text generation datasets, six knowledge and question answering tasks, and one mathematical reasoning task. We present results comparing the performance of prompting LLMs to fine-tuned baselines based on BERT and T5-style models. Our results suggest large gaps in performance between high-resource languages, such as English, and African languages across most tasks; but performance also varies based on the availability of monolingual data resources. Our findings confirm that performance on African languages continues to remain a hurdle for current LLMs, underscoring the need for additional efforts to close this gap. https://mcgill-nlp.github.io/AfroBench/