# Africa-Centric Self-Supervised Pre-Training for Multilingual Speech Representation in a Sub-Saharan Context

arXiv: https://arxiv.org/abs/2404.02000 · HTML: https://mlsanity.com/paper/2404.02000/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 17 of 17)

- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training](https://mlsanity.com/paper/2108.06209/) (2021)
- [VoxLingua107: a Dataset for Spoken Language Recognition](https://mlsanity.com/paper/2011.12998/) (2020)
- [AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages](https://mlsanity.com/paper/2211.03263/) (2022)
- [SERENGETI: Massively Multilingual Language Models for Africa](https://mlsanity.com/paper/2212.10785/) (2022)
- [AfriSpeech-200: Pan-African Accented Speech Dataset for Clinical and General Domain ASR](https://mlsanity.com/paper/2310.00274/) (2023)
- [A Survey of Multilingual Models for Automatic Speech Recognition](https://mlsanity.com/paper/2202.12576/) (2022)
- [BembaSpeech: A Speech Recognition Corpus for the Bemba Language](https://mlsanity.com/paper/2102.04889/) (2021)
- [Speech Resources in the Tamasheq Language](https://mlsanity.com/paper/2201.05051/) (2022)
- [KenSwQuAD -- A Question Answering Dataset for Swahili Low Resource Language](https://mlsanity.com/paper/2205.02364/) (2022)
- [Large vocabulary speech recognition for languages of Africa: multilingual modeling and self-supervised learning](https://mlsanity.com/paper/2208.03067/) (2022)

## Built on by (most cited first, 8 of 8)

- [AfriHuBERT: A self-supervised speech representation model for African languages](https://mlsanity.com/paper/2409.20201/) (2024)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Automatic Speech Recognition for African Low-Resource Languages: Challenges and Future Directions](https://mlsanity.com/paper/2505.11690/) (2025)
- [Voice of a Continent: Mapping Africa's Speech Technology Frontier](https://mlsanity.com/paper/2505.18436/) (2025)
- [Speech Language Models for Under-Represented Languages: Insights from Wolof](https://mlsanity.com/paper/2509.15362/) (2025)
- [Automatic Speech Recognition (ASR) for African Low-Resource Languages: A Systematic Literature Review](https://mlsanity.com/paper/2510.01145/) (2025)
- [Scaling HuBERT for African Languages: From Base to Large and XL](https://mlsanity.com/paper/2511.23370/) (2025)
- [Opportunities and Challenges of Natural Language Processing for Low-Resource Senegalese Languages in Social Science Research](https://mlsanity.com/paper/2601.09716/) (2025)

## Abstract (arXiv)

We present the first self-supervised multilingual speech model trained exclusively on African speech. The model learned from nearly 60 000 hours of unlabeled speech segments in 21 languages and dialects spoken in sub-Saharan Africa. On the SSA subset of the FLEURS-102 dataset, our approach based on a HuBERTbase (0.09B) architecture shows competitive results, for ASR downstream task, compared to the w2v-bert-51 (0.6B) pre-trained model proposed in the FLEURS benchmark, while being more efficient by using 7x less data and 6x less parameters. Furthermore, in the context of a LID downstream task, our approach outperforms FLEURS baselines accuracy by over 22%.