# AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages

arXiv: https://arxiv.org/abs/2211.03263 · HTML: https://mlsanity.com/paper/2211.03263/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 7 Nov 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [Deep Bayesian Active Learning for Natural Language Processing: Results of a Large-Scale Empirical Study](https://mlsanity.com/paper/1808.05697/) (2018)
- [Less is more: sampling chemical space with active learning](https://mlsanity.com/paper/1801.09319/) (2018)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [Coherence boosting: When your pretrained language model is not paying enough attention](https://mlsanity.com/paper/2110.08294/) (2021)
- [KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi](https://mlsanity.com/paper/2010.12174/) (2020)
- [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022)
- [yosm: A new yoruba sentiment corpus for movie reviews](https://mlsanity.com/paper/2204.09711/) (2022)
- [Multilingual Detection of Personal Employment Status on Twitter](https://mlsanity.com/paper/2203.09178/) (2022)

## Built on by (most cited first, 30 of 46)

- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [SERENGETI: Massively Multilingual Language Models for Africa](https://mlsanity.com/paper/2212.10785/) (2022)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages](https://mlsanity.com/paper/2305.13989/) (2023)
- [Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding](https://mlsanity.com/paper/2412.17837/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Cheetah: Natural Language Generation for 517 African Languages](https://mlsanity.com/paper/2401.01053/) (2024)
- [NLNDE at SemEval-2023 Task 12: Adaptive Pretraining and Source Language Selection for Low-Resource Multilingual Sentiment Analysis](https://mlsanity.com/paper/2305.00090/) (2023)
- [Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia](https://mlsanity.com/paper/2503.07920/) (2025)
- [How Good are Commercial Large Language Models on African Languages?](https://mlsanity.com/paper/2305.06530/) (2023)
- [EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation](https://mlsanity.com/paper/2403.13737/) (2024)
- [Africa-Centric Self-Supervised Pre-Training for Multilingual Speech Representation in a Sub-Saharan Context](https://mlsanity.com/paper/2404.02000/) (2024)
- [Ethio-Fake: Cutting-Edge Approaches to Combat Fake News in Under-Resourced Languages Using Explainable AI](https://mlsanity.com/paper/2410.02609/) (2024)
- [The State of Large Language Models for African Languages: Progress and Challenges](https://mlsanity.com/paper/2506.02280/) (2025)
- [Several categories of Large Language Models (LLMs): A Short Survey](https://mlsanity.com/paper/2307.10188/) (2023)
- [State of NLP in Kenya: A Survey](https://mlsanity.com/paper/2410.09948/) (2024)
- [PuoBERTa: Training and evaluation of a curated language model for Setswana](https://mlsanity.com/paper/2310.09141/) (2023)
- [Exploring Boundaries and Intensities in Offensive and Hate Speech: Unveiling the Complex Spectrum of Social Media Discourse](https://mlsanity.com/paper/2404.12042/) (2024)
- [UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages](https://mlsanity.com/paper/2406.09717/) (2024)
- [AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text](https://mlsanity.com/paper/2503.18247/) (2025)
- [Adapting to the Low-Resource Double-Bind: Investigating Low-Compute Methods on Low-Resource African Languages](https://mlsanity.com/paper/2303.16985/) (2023)
- [Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding](https://mlsanity.com/paper/2507.09758/) (2025)
- [GMNLP at SemEval-2023 Task 12: Sentiment Analysis with Phylogeny-Based Adapters](https://mlsanity.com/paper/2304.12979/) (2023)
- [The Less the Merrier? Investigating Language Representation in Multilingual Models](https://mlsanity.com/paper/2310.13228/) (2023)
- [Sentiment Analysis Across Multiple African Languages: A Current Benchmark](https://mlsanity.com/paper/2310.14120/) (2023)
- [ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model](https://mlsanity.com/paper/2404.02534/) (2024)
- [From N-grams to Pre-trained Multilingual Models For Language Identification](https://mlsanity.com/paper/2410.08728/) (2024)
- [Bridging Gaps in Natural Language Processing for Yor`ub'a: A Systematic Review of a Decade of Progress and Prospects](https://mlsanity.com/paper/2502.17364/) (2025)
- [KinyaColBERT: A Lexically Grounded Retrieval Model for Low-Resource Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.03241/) (2025)
- [FonMTL: Towards Multitask Learning for the Fon Language](https://mlsanity.com/paper/2308.14280/) (2023)

## Abstract (arXiv)

In recent years, multilingual pre-trained language models have gained prominence due to their remarkable performance on numerous downstream Natural Language Processing tasks (NLP). However, pre-training these large multilingual language models requires a lot of training data, which is not available for African Languages. Active learning is a semi-supervised learning algorithm, in which a model consistently and dynamically learns to identify the most beneficial samples to train itself on, in order to achieve better optimization and performance on downstream tasks. Furthermore, active learning effectively and practically addresses real-world data scarcity. Despite all its benefits, active learning, in the context of NLP and especially multilingual language models pretraining, has received little consideration. In this paper, we present AfroLM, a multilingual language model pretrained from scratch on 23 African languages (the largest effort to date) using our novel self-active learning framework. Pretrained on a dataset significantly (14x) smaller than existing baselines, AfroLM outperforms many multilingual pretrained language models (AfriBERTa, XLMR-base, mBERT) on various NLP downstream tasks (NER, text classification, and sentiment analysis). Additional out-of-domain sentiment analysis experiments show that AfroLM is able to generalize well across various domains. We release the code source, and our datasets used in our framework at https://github.com/bonaventuredossou/MLMAL.