# SERENGETI: Massively Multilingual Language Models for Africa

arXiv: https://arxiv.org/abs/2212.10785 · HTML: https://mlsanity.com/paper/2212.10785/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 128 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 30 of 45)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Multilingual Constituency Parsing with Self-Attention and Pre-Training](https://mlsanity.com/paper/1812.11760/) (2018)
- [Fast WordPiece Tokenization](https://mlsanity.com/paper/2012.15524/) (2020)
- [Larger-Scale Transformers for Multilingual Masked Language Modeling](https://mlsanity.com/paper/2105.00572/) (2021)
- [XLM-E: Cross-lingual Language Model Pre-training via ELECTRA](https://mlsanity.com/paper/2106.16138/) (2021)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)

## Built on by (most cited first, 30 of 40)

- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [CLIcK: A Benchmark Dataset of Cultural and Linguistic Intelligence in Korean](https://mlsanity.com/paper/2403.06412/) (2024)
- [IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models](https://mlsanity.com/paper/2406.03368/) (2024)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [MaLA-500: Massive Language Adaptation of Large Language Models](https://mlsanity.com/paper/2401.13303/) (2024)
- [Taxi1500: A Multilingual Dataset for Text Classification in 1500 Languages](https://mlsanity.com/paper/2305.08487/) (2023)
- [GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages](https://mlsanity.com/paper/2410.23825/) (2024)
- [An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)](https://mlsanity.com/paper/2503.10267/) (2025)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Where Are We? Evaluating LLM Performance on African Languages](https://mlsanity.com/paper/2502.19582/) (2025)
- [How Good are Commercial Large Language Models on African Languages?](https://mlsanity.com/paper/2305.06530/) (2023)
- [EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation](https://mlsanity.com/paper/2403.13737/) (2024)
- [Africa-Centric Self-Supervised Pre-Training for Multilingual Speech Representation in a Sub-Saharan Context](https://mlsanity.com/paper/2404.02000/) (2024)
- [Lugha-Llama: Adapting Large Language Models for African Languages](https://mlsanity.com/paper/2504.06536/) (2025)
- [State of NLP in Kenya: A Survey](https://mlsanity.com/paper/2410.09948/) (2024)
- [PuoBERTa: Training and evaluation of a curated language model for Setswana](https://mlsanity.com/paper/2310.09141/) (2023)
- [Ibom NLP: A Step Toward Inclusive Natural Language Processing for Nigeria's Minority Languages](https://mlsanity.com/paper/2511.06531/) (2025)
- [A Systematic Review of NLP for Ghanaian Languages: Datasets, Models, and a Research Roadmap](https://mlsanity.com/paper/2405.06818/) (2024)
- [NaijaRC: A Multi-choice Reading Comprehension Dataset for Nigerian Languages](https://mlsanity.com/paper/2308.09768/) (2023)
- [Voice of a Continent: Mapping Africa's Speech Technology Frontier](https://mlsanity.com/paper/2505.18436/) (2025)
- [TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models](https://mlsanity.com/paper/2401.06620/) (2024)
- [Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval](https://mlsanity.com/paper/2505.19356/) (2025)
- [Can Embedding Similarity Predict Cross-Lingual Transfer? A Systematic Study on African Languages](https://mlsanity.com/paper/2601.03168/) (2026)
- [ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model](https://mlsanity.com/paper/2404.02534/) (2024)
- [From N-grams to Pre-trained Multilingual Models For Language Identification](https://mlsanity.com/paper/2410.08728/) (2024)
- [Bridging Gaps in Natural Language Processing for Yor`ub'a: A Systematic Review of a Decade of Progress and Prospects](https://mlsanity.com/paper/2502.17364/) (2025)
- [Improving Multilingual Math Reasoning for African Languages](https://mlsanity.com/paper/2505.19848/) (2025)

## Abstract (arXiv)

Multilingual pretrained language models (mPLMs) acquire valuable, generalizable linguistic information during pretraining and have advanced the state of the art on task-specific finetuning. To date, only ~31 out of ~2,000 African languages are covered in existing language models. We ameliorate this limitation by developing SERENGETI, a massively multilingual language model that covers 517 African languages and language varieties. We evaluate our novel models on eight natural language understanding tasks across 20 datasets, comparing to 4 mPLMs that cover 4-23 African languages. SERENGETI outperforms other models on 11 datasets across the eights tasks, achieving 82.27 average F1. We also perform analyses of errors from our models, which allows us to investigate the influence of language genealogy and linguistic similarity when the models are applied under zero-shot settings. We will publicly release our models for research.footnotehrefhttps://github.com/UBC-NLP/serengetihttps://github.com/UBC-NLP/serengeti