# InkubaLM: A small language model for low-resource African languages

arXiv: https://arxiv.org/abs/2408.17024 · HTML: https://mlsanity.com/paper/2408.17024/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 30 Aug 2024.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [XNLI: Evaluating Cross-lingual Sentence Representations](https://mlsanity.com/paper/1809.05053/) (2018)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023)
- [LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model](https://mlsanity.com/paper/2401.02330/) (2024)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages](https://mlsanity.com/paper/2302.08956/) (2023)
- [NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysis](https://mlsanity.com/paper/2201.08277/) (2022)
- [Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages](https://mlsanity.com/paper/2010.03179/) (2020)
- [Masakhane -- Machine Translation For Africa](https://mlsanity.com/paper/2003.11529/) (2020)
- [SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)](https://mlsanity.com/paper/2304.06845/) (2023)
- [The Effect of Domain and Diacritics in Yor`ub'a-English Neural Machine Translation](https://mlsanity.com/paper/2103.08647/) (2021)
- [MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT](https://mlsanity.com/paper/2402.16840/) (2024)
- [Small Language Models are Good Too: An Empirical Study of Zero-Shot Classification](https://mlsanity.com/paper/2404.11122/) (2024)
- [Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistance](https://mlsanity.com/paper/2405.17076/) (2024)
- [When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models](https://mlsanity.com/paper/2404.08634/) (2024)
- [Small-E: Small Language Model with Linear Attention for Efficient Speech Synthesis](https://mlsanity.com/paper/2406.04467/) (2024)
- [Achieving Sparse Activation in Small Language Models](https://mlsanity.com/paper/2406.06562/) (2024)
- [Can Small Language Models Learn, Unlearn, and Retain Noise Patterns?](https://mlsanity.com/paper/2407.00996/) (2024)

## Built on by (most cited first, 18 of 18)

- [Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research](https://mlsanity.com/paper/2412.04497/) (2024)
- [Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models](https://mlsanity.com/paper/2410.01335/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect](https://mlsanity.com/paper/2409.17912/) (2024)
- [AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages](https://mlsanity.com/paper/2501.08284/) (2025)
- [Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures](https://mlsanity.com/paper/2510.24081/) (2025)
- [The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs](https://mlsanity.com/paper/2505.18356/) (2025)
- [Lugha-Llama: Adapting Large Language Models for African Languages](https://mlsanity.com/paper/2504.06536/) (2025)
- [Lillama: Large Language Models Compression via Low-Rank Feature Distillation](https://mlsanity.com/paper/2412.16719/) (2024)
- [HausaNLP: Current Status, Challenges and Future Directions for Hausa Natural Language Processing](https://mlsanity.com/paper/2505.14311/) (2025)
- [Evaluating LLMs for Career Guidance: Comparative Analysis of Computing Competency Recommendations Across Ten African Countries](https://mlsanity.com/paper/2510.18902/) (2025)
- [LLMs and Agentic AI in Insurance Decision-Making: Opportunities and Challenges For Africa](https://mlsanity.com/paper/2508.15110/) (2025)
- [Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Models](https://mlsanity.com/paper/2510.07203/) (2025)
- [MzansiText and MzansiLM: An Open Corpus and Decoder-Only Language Model for South African Languages](https://mlsanity.com/paper/2603.20732/) (2026)
- [A Survey of Text and Speech Resources for Hausa and Fongbe: Availability, Quality, and Gaps for NLP Development](https://mlsanity.com/paper/2605.22828/) (2026)
- [From Script to Semantics: Prompting Strategies for African NLI](https://mlsanity.com/paper/2606.03304/) (2026)
- [From Speech to Text Corpora: Evaluating ASR-Based Data Acquisition for Low-Resource Fongbe and Hausa](https://mlsanity.com/paper/2606.22274/) (2026)
- [Towards Model as a Library: Offline, Community-Sourced AI for Low-Resource African Languages](https://mlsanity.com/paper/2609.38574/) (2026)

## Abstract (arXiv)

High-resource language models often fall short in the African context, where there is a critical need for models that are efficient, accessible, and locally relevant, even amidst significant computing and data constraints. This paper introduces InkubaLM, a small language model with 0.4 billion parameters, which achieves performance comparable to models with significantly larger parameter counts and more extensive training data on tasks such as machine translation, question-answering, AfriMMLU, and the AfriXnli task. Notably, InkubaLM outperforms many larger models in sentiment analysis and demonstrates remarkable consistency across multiple languages. This work represents a pivotal advancement in challenging the conventional paradigm that effective language models must rely on substantial resources. Our model and datasets are publicly available at https://huggingface.co/lelapa to encourage research and development on low-resource languages.