# PuoBERTa: Training and evaluation of a curated language model for Setswana

arXiv: https://arxiv.org/abs/2310.09141 · HTML: https://mlsanity.com/paper/2310.09141/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 39 references that are in the map.
- First posted on arXiv 13 Oct 2023.

## Builds on (most cited first, 25 of 25)

- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Beyond English-Centric Multilingual Machine Translation](https://mlsanity.com/paper/2010.11125/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [Neural Machine Translation for Low-Resource Languages: A Survey](https://mlsanity.com/paper/2106.15115/) (2021)
- [Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages](https://mlsanity.com/paper/2212.05409/) (2022)
- [A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation](https://mlsanity.com/paper/2205.02022/) (2022)
- [Survey of Low-Resource Machine Translation](https://mlsanity.com/paper/2109.00486/) (2021)
- [What Language Model to Train if You Have One Million GPU Hours?](https://mlsanity.com/paper/2210.15424/) (2022)
- [AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages](https://mlsanity.com/paper/2211.03263/) (2022)
- [SERENGETI: Massively Multilingual Language Models for Africa](https://mlsanity.com/paper/2212.10785/) (2022)
- [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://mlsanity.com/paper/2210.12391/) (2022)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalan](https://mlsanity.com/paper/2107.07903/) (2021)
- [MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languages](https://mlsanity.com/paper/2305.13989/) (2023)
- [Using Monolingual Data in Neural Machine Translation: a Systematic Study](https://mlsanity.com/paper/1903.11437/) (2019)
- [Investigating an approach for low resource language dataset creation, curation and classification: Setswana and Sepedi](https://mlsanity.com/paper/2003.04986/) (2020)
- [Adapting Monolingual Models: Data can be Scarce when Language Similarity is High](https://mlsanity.com/paper/2105.02855/) (2021)
- [Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corpora](https://mlsanity.com/paper/2303.03750/) (2023)
- [Subword Segmental Language Modelling for Nguni Languages](https://mlsanity.com/paper/2210.06525/) (2022)
- [You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models](https://mlsanity.com/paper/2210.07135/) (2022)
- [On the ability of monolingual models to learn language-agnostic representations](https://mlsanity.com/paper/2109.01942/) (2021)
- [When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scale](https://mlsanity.com/paper/2305.14124/) (2023)

## Built on by (most cited first, 6 of 6)

- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Pula: Training Large Language Models for Setswana](https://mlsanity.com/paper/2408.02239/) (2024)
- [The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages](https://mlsanity.com/paper/2511.09197/) (2025)
- [Flick: Few Labels Text Classification using K-Aware Intermediate Learning in Multi-Task Low-Resource Languages](https://mlsanity.com/paper/2506.10292/) (2025)
- [Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Review](https://mlsanity.com/paper/2407.00252/) (2024)
- [LLMs and Agentic AI in Insurance Decision-Making: Opportunities and Challenges For Africa](https://mlsanity.com/paper/2508.15110/) (2025)

## Abstract (arXiv)

Natural language processing (NLP) has made significant progress for well-resourced languages such as English but lagged behind for low-resource languages like Setswana. This paper addresses this gap by presenting PuoBERTa, a customised masked language model trained specifically for Setswana. We cover how we collected, curated, and prepared diverse monolingual texts to generate a high-quality corpus for PuoBERTa's training. Building upon previous efforts in creating monolingual resources for Setswana, we evaluated PuoBERTa across several NLP tasks, including part-of-speech (POS) tagging, named entity recognition (NER), and news categorisation. Additionally, we introduced a new Setswana news categorisation dataset and provided the initial benchmarks using PuoBERTa. Our work demonstrates the efficacy of PuoBERTa in fostering NLP capabilities for understudied languages like Setswana and paves the way for future research directions.