mlsanityData to 7 Oct 2026

IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models

arXiv 2406.03368 ↗First posted 5 Jun 2024cs.CL cs.AIcited by 43

Not on the map. Why?

Abstract (arXiv)

Despite the widespread adoption of Large language models (LLMs), their remarkable capabilities remain limited to a few high-resource languages. Additionally, many low-resource languages (eg African languages) are often evaluated only on basic text classification tasks due to the lack of appropriate or comprehensive benchmarks outside of high-resource languages. In this paper, we introduce IrokoBench -- a human-translated benchmark dataset for 17 typologically-diverse low-resource African languages covering three tasks: natural language inference~(AfriXNLI), mathematical reasoning~(AfriMGSM), and multi-choice knowledge-based question answering~(AfriMMLU). We use IrokoBench to evaluate zero-shot, few-shot, and translate-test settings~(where test sets are translated into English) across 10 open and six proprietary LLMs. Our evaluation reveals a significant performance gap between high-resource languages~(such as English and French) and low-resource African languages. We observe a significant performance gap between open and proprietary models, with the highest performing open model, Gemma 2 27B only at 63% of the best-performing proprietary model GPT-4o performance. In addition, machine translating the test set to English before evaluation helped to close the gap for larger models that are English-centric, such as Gemma 2 27B and LLaMa 3.1 70B. These findings suggest that more efforts are needed to develop and adapt LLMs for African languages.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 49 of its 69 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

49 of 69 references.

References · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
37 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  3. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  5. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  6. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  7. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  8. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 518
  9. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 344
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  11. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  12. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  13. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  14. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  15. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 203
  16. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  17. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  18. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  19. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  20. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  21. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  22. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 149
  23. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  24. 2021Gender Bias in Machine Translationcited by 116
  25. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  26. 2021What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpuscited by 79
  27. 2020Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpuscited by 76
  28. 2021Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translationcited by 70
  29. 2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
  30. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  31. 2023AfroBench: How Good are Large Language Models on African Languages?cited by 43
  32. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  33. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  34. 2022Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation?cited by 24
  35. 2023Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languagescited by 23
  36. 2024Predicting Machine Translation Performance on Low-Resource Languages: The Role of Domain Similaritycited by 15
  37. 2023Measuring Sentiment Bias in Machine Translationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 22.

43 papers.

Sort:

By area:

  1. 2026Tiny Aya: Bridging Scale and Multilingual Depthcited by 14
  2. 2025When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracycited by 13
  3. 2025The Token Tax: Systematic Bias in Multilingual Tokenizationcited by 13
  4. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  5. 2025Lugha-Llama: Adapting Large Language Models for African Languagescited by 8
  6. 2025MazeEval: A Benchmark for Testing Sequential Decision-Making in Language Modelscited by 8
  7. 2024Uhura: A Benchmark for Evaluating Scientific Question Answering and Truthfulness in Low-Resource African Languagescited by 7
  8. 2025mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Taskscited by 7
  9. 2025Voice of a Continent: Mapping Africa's Speech Technology Frontiercited by 4
  10. 2026Evaluating and Benchmarking the System One Model Jevcited by 3
  11. 2025LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriamacited by 2
  12. 2025MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoningcited by 2
31 more
  1. 2025AI Diffusion in Low Resource Language Countriescited by 2
  2. 2026Afri-MCQA: Multimodal Cultural Question Answering for African Languagescited by 2
  3. 2026MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluationcited by 2
  4. 2026AfriNLLB: Efficient Translation Models for African Languagescited by 2
  5. 2025Sentiment Analysis on the young people's perception about the mobile Internet costs in Senegalcited by 1
  6. 2025Improving Multilingual Math Reasoning for African Languagescited by 1
  7. 2024Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustmentscited by 0
  8. 2025NaijaNLP: A Survey of Nigerian Low-Resource Languagescited by 0
  9. 2025FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Modelscited by 0
  10. 2025Against 'softmaxing' culturecited by 0
  11. 2025The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLPcited by 0
  12. 2025Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Modelscited by 0
  13. 2025Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languagescited by 0
  14. 2025Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Researchcited by 0
  15. 2026BYOL: Bring Your Own Language Into LLMscited by 0
  16. 2026AfroScope: A Framework for Studying the Linguistic Landscape of Africacited by 0
  17. 2026Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspectivecited by 0
  18. 2026MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languagescited by 0
  19. 2026SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Palicited by 0
  20. 2026RoMathExam: A Longitudinal Dataset of Romanian Math Exams (1895-2025) with a Seven-Decade Core (1957-2025)cited by 0
  21. 2026AfriVox-v2: A Domain-Verticalized Benchmark for In-the-Wild African Speech Recognitioncited by 0
  22. 2026The Annotation Scarcity Paradox in Low-Resource NLP Evaluation: A Decade of Acceleration and Emerging Constraintscited by 0
  23. 2026A Survey of Text and Speech Resources for Hausa and Fongbe: Availability, Quality, and Gaps for NLP Developmentcited by 0
  24. 2026From Script to Semantics: Prompting Strategies for African NLIcited by 0
  25. 2026RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfercited by 0
  26. 2026Position: AI Leaderboards Are Underserving the Global South: A Case Study from Indiacited by 0
  27. 2026TranslatePsy-AfriSLM: High-Quality Data Scaling For Low-Resource Machine Translationcited by 0
  28. 2026Structure, Association, and Decision Value: Representation-Based Difficulty Estimation for Adaptive Inference in African-Language NLIcited by 0
  29. 2026The Geometry of Low-Resource Language Representationscited by 0
  30. 2026AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Contentcited by 0
  31. 2026GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
  2. 2025AfriMTEB and AfriE5: Benchmarking and Adapting Text Embedding Models for African Languages
  3. 2024Linguini: A benchmark for language-agnostic linguistic reasoning
  4. 2024Evaluating Multilingual Long-Context Models for Retrieval and Reasoning
  5. 2025Mafoko: Structuring and Building Open Multilingual Terminologies for South African NLP
  6. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  7. 2025LingBench++: A Linguistically-Informed Benchmark and Reasoning Framework for Multi-Step and Cross-Cultural Inference with LLMs
  8. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models