mlsanityData to 7 Oct 2026

PuoBERTa: Training and evaluation of a curated language model for Setswana

arXiv 2310.09141 ↗First posted 13 Oct 2023cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Natural language processing (NLP) has made significant progress for well-resourced languages such as English but lagged behind for low-resource languages like Setswana. This paper addresses this gap by presenting PuoBERTa, a customised masked language model trained specifically for Setswana. We cover how we collected, curated, and prepared diverse monolingual texts to generate a high-quality corpus for PuoBERTa's training. Building upon previous efforts in creating monolingual resources for Setswana, we evaluated PuoBERTa across several NLP tasks, including part-of-speech (POS) tagging, named entity recognition (NER), and news categorisation. Additionally, we introduced a new Setswana news categorisation dataset and provided the initial benchmarks using PuoBERTa. Our work demonstrates the efficacy of PuoBERTa in fostering NLP capabilities for understudied languages like Setswana and paves the way for future research directions.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 13 Oct 2023.

Builds on

25 of 39 references.

References · 25

  1. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  3. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  4. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  5. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  6. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  7. 2021Neural Machine Translation for Low-Resource Languages: A Surveycited by 108
  8. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 99
  9. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 97
  10. 2021Survey of Low-Resource Machine Translationcited by 92
  11. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  12. 2022AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languagescited by 46
13 more
  1. 2022SERENGETI: Massively Multilingual Language Models for Africacited by 40
  2. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  3. 2023MasakhaNEWS: News Topic Classification for African languagescited by 30
  4. 2021Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalancited by 26
  5. 2023MasakhaPOS: Part-of-Speech Tagging for Typologically Diverse African Languagescited by 23
  6. 2019Using Monolingual Data in Neural Machine Translation: a Systematic Studycited by 21
  7. 2020Investigating an approach for low resource language dataset creation, curation and classification: Setswana and Sepedicited by 15
  8. 2021Adapting Monolingual Models: Data can be Scarce when Language Similarity is Highcited by 12
  9. 2023Preparing the Vuk'uzenzele and ZA-gov-multilingual South African multilingual corporacited by 12
  10. 2022Subword Segmental Language Modelling for Nguni Languagescited by 8
  11. 2022You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Modelscited by 6
  12. 2021On the ability of monolingual models to learn language-agnostic representationscited by 5
  13. 2023When Does Monolingual Data Help Multilingual Translation: The Role of Domain and Model Scalecited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 4.

6 papers.

  1. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  2. 2024Pula: Training Large Language Models for Setswanacited by 2
  3. 2025The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languagescited by 2
  4. 2025Flick: Few Labels Text Classification using K-Aware Intermediate Learning in Multi-Task Low-Resource Languagescited by 1
  5. 2024Assistive Image Annotation Systems with Deep Learning and Natural Language Capabilities: A Reviewcited by 0
  6. 2025LLMs and Agentic AI in Insurance Decision-Making: Opportunities and Challenges For Africacited by 0