mlsanityData to 14 Oct 2026

Lugha-Llama: Adapting Large Language Models for African Languages

arXiv 2504.06536 ↗First posted 9 Apr 2025cs.CL cs.AI cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have achieved impressive results in a wide range of natural language applications. However, they often struggle to recognize low-resource languages, in particular African languages, which are not well represented in large training corpora. In this paper, we consider how to adapt LLMs to low-resource African languages. We find that combining curated data from African languages with high-quality English educational texts results in a training mix that substantially improves the model's performance on these languages. On the challenging IrokoBench dataset, our models consistently achieve the best performance amongst similarly sized baselines, particularly on knowledge-intensive multiple-choice questions (AfriMMLU). Additionally, on the cross-lingual question answering benchmark AfriQA, our models outperform the base model by over 10%. To better understand the role of English data during training, we translate a subset of 200M tokens into Swahili language and perform an analysis which reveals that the content of these data is primarily responsible for the strong performance. We release our models and data to encourage future research on African languages.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 9 Apr 2025.

Builds on

42 of 56 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
30 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  2. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  3. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  6. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  7. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  8. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  9. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  10. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  11. 2024A Survey on Data Selection for Language Modelscited by 236
  12. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  13. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  14. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  15. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 165
  16. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  17. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  18. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  19. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  20. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
  21. 2024Adapting Large Language Models for Document-Level Machine Translationcited by 59
  22. 2024LLaMAX: Scaling Linguistic Horizons of LLM by Enhancing Translation Capabilities Beyond 100 Languagescited by 48
  23. 2024IrokoBench: A New Benchmark for African Languages in the Age of Large Language Modelscited by 43
  24. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 42
  25. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  26. 2021Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translationcited by 29
  27. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  28. 2023MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languagescited by 23
  29. 2024InkubaLM: A small language model for low-resource African languagescited by 18
  30. 2023How Good are Commercial Large Language Models on African Languages?cited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2026AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languagescited by 12
  2. 2025FilBench: Can LLMs Understand and Generate Filipino?cited by 3
  3. 2025MERLIN: Multi-Stage Curriculum Alignment for Multilingual Encoder-LLM Integration in Cross-Lingual Reasoningcited by 2
  4. 2025Improving Multilingual Math Reasoning for African Languagescited by 1
  5. 2025Towards culturally-appropriate conversational AI for health in the majority world: An exploratory study with citizens and professionals in Latin Americacited by 1
  6. 2025Agentic Educational Content Generation for African Languages on Edge Devicescited by 1
  7. 2026AfriSUD: A Dependency Treebank Collection for Evaluating Models on African Languagescited by 1
  8. 2026MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languagescited by 0