# Transfer Learning and Distant Supervision for Multilingual Transformer Models: A Study on African Languages

arXiv: https://arxiv.org/abs/2010.03179 · HTML: https://mlsanity.com/paper/2010.03179/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Oct 2020.

## Built on by (most cited first, 30 of 52)

- [XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation](https://mlsanity.com/paper/2104.07412/) (2021)
- [MasakhaNER: Named Entity Recognition for African Languages](https://mlsanity.com/paper/2103.11811/) (2021)
- [Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuning](https://mlsanity.com/paper/2204.06487/) (2022)
- [BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer](https://mlsanity.com/paper/2305.14857/) (2023)
- [AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages](https://mlsanity.com/paper/2211.03263/) (2022)
- [SERENGETI: Massively Multilingual Language Models for Africa](https://mlsanity.com/paper/2212.10785/) (2022)
- [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://mlsanity.com/paper/2210.12391/) (2022)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [MasakhaNEWS: News Topic Classification for African languages](https://mlsanity.com/paper/2304.09972/) (2023)
- [Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance](https://mlsanity.com/paper/2311.01108/) (2023)
- [AfriQA: Cross-lingual Open-Retrieval Question Answering for African Languages](https://mlsanity.com/paper/2305.06897/) (2023)
- [Is BERT Robust to Label Noise? A Study on Learning with Noisy Labels in Text Classification](https://mlsanity.com/paper/2204.09371/) (2022)
- [InkubaLM: A small language model for low-resource African languages](https://mlsanity.com/paper/2408.17024/) (2024)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead](https://mlsanity.com/paper/2505.21315/) (2025)
- [Analysing the Noise Model Error for Realistic Noisy Label Data](https://mlsanity.com/paper/2101.09763/) (2021)
- [Label-Descriptive Patterns and Their Application to Characterizing Classification Errors](https://mlsanity.com/paper/2110.09599/) (2021)
- [Traffic-Domain Video Question Answering with Automatic Captioning](https://mlsanity.com/paper/2307.09636/) (2023)
- [Where Are We? Evaluating LLM Performance on African Languages](https://mlsanity.com/paper/2502.19582/) (2025)
- [How Good are Commercial Large Language Models on African Languages?](https://mlsanity.com/paper/2305.06530/) (2023)
- [Por Qu'e N~ao Utiliser Alla Spraak? Mixed Training with Gradient Optimization in Few-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2204.13869/) (2022)
- [Meta Self-Refinement for Robust Learning with Weak Supervision](https://mlsanity.com/paper/2205.07290/) (2022)
- [Data-Efficient Strategies for Expanding Hate Speech Detection into Under-Resourced Languages](https://mlsanity.com/paper/2210.11359/) (2022)
- [To Translate or Not to Translate: A Systematic Investigation of Translation-Based Cross-Lingual Transfer to Low-Resource Languages](https://mlsanity.com/paper/2311.09404/) (2023)
- [The first large scale collection of diverse Hausa language datasets](https://mlsanity.com/paper/2102.06991/) (2021)
- [Training Cross-Lingual embeddings for Setswana and Sepedi](https://mlsanity.com/paper/2111.06230/) (2021)
- [Unraveling Social Perceptions & Behaviors towards Migrants on Twitter](https://mlsanity.com/paper/2112.06642/) (2021)
- [Learning Action Conditions from Instructional Manuals for Instruction Understanding](https://mlsanity.com/paper/2205.12420/) (2022)
- [PInKS: Preconditioned Commonsense Inference with Minimal Supervision](https://mlsanity.com/paper/2206.07920/) (2022)
- [Prompt Learning to Mitigate Catastrophic Forgetting in Cross-lingual Transfer for Open-domain Dialogue Generation](https://mlsanity.com/paper/2305.07393/) (2023)

## Abstract (arXiv)

Multilingual transformer models like mBERT and XLM-RoBERTa have obtained great improvements for many NLP tasks on a variety of languages. However, recent works also showed that results from high-resource languages could not be easily transferred to realistic, low-resource scenarios. In this work, we study trends in performance for different amounts of available resources for the three African languages Hausa, isiXhosa and Yor`ub'a on both NER and topic classification. We show that in combination with transfer learning or distant supervision, these models can achieve with as little as 10 or 100 labeled sentences the same performance as baselines with much more supervised training data. However, we also find settings where this does not hold. Our discussions and additional experiments on assumptions such as time and hardware restrictions highlight challenges and opportunities in low-resource learning.