# Cross-lingual transfer of multilingual models on low resource African Languages

arXiv: https://arxiv.org/abs/2409.10965 · HTML: https://mlsanity.com/paper/2409.10965/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 22 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages](https://mlsanity.com/paper/2302.08956/) (2023)
- [KINNEWS and KIRNEWS: Benchmarking Cross-Lingual Text Classification for Kinyarwanda and Kirundi](https://mlsanity.com/paper/2010.12174/) (2020)
- [Low Resource Neural Machine Translation: A Benchmark for Five African Languages](https://mlsanity.com/paper/2003.14402/) (2020)
- [Cross-lingual Transfer of Monolingual Models](https://mlsanity.com/paper/2109.07348/) (2021)
- [Low-Resource Language Modelling of South African Languages](https://mlsanity.com/paper/2104.00772/) (2021)
- [Universal Cross-Lingual Text Classification](https://mlsanity.com/paper/2406.11028/) (2024)

## Built on by (most cited first, 5 of 5)

- [Optimism, Expectation, or Sarcasm? Multi-Class Hope Speech Detection in Spanish and English](https://mlsanity.com/paper/2504.17974/) (2025)
- [How does a Multilingual LM Handle Multiple Languages?](https://mlsanity.com/paper/2502.04269/) (2025)
- [MAGE: Multi-Head Attention Guided Embeddings for Low Resource Sentiment Classification](https://mlsanity.com/paper/2502.17987/) (2025)
- [AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages](https://mlsanity.com/paper/2502.18020/) (2025)
- [textscCantoNLU: A benchmark for Cantonese natural language understanding](https://mlsanity.com/paper/2510.20670/) (2025)

## Abstract (arXiv)

Large multilingual models have significantly advanced natural language processing (NLP) research. However, their high resource demands and potential biases from diverse data sources have raised concerns about their effectiveness across low-resource languages. In contrast, monolingual models, trained on a single language, may better capture the nuances of the target language, potentially providing more accurate results. This study benchmarks the cross-lingual transfer capabilities from a high-resource language to a low-resource language for both, monolingual and multilingual models, focusing on Kinyarwanda and Kirundi, two Bantu languages. We evaluate the performance of transformer based architectures like Multilingual BERT (mBERT), AfriBERT, and BantuBERTa against neural-based architectures such as BiGRU, CNN, and char-CNN. The models were trained on Kinyarwanda and tested on Kirundi, with fine-tuning applied to assess the extent of performance improvement and catastrophic forgetting. AfriBERT achieved the highest cross-lingual accuracy of 88.3% after fine-tuning, while BiGRU emerged as the best-performing neural model with 83.3% accuracy. We also analyze the degree of forgetting in the original language post-fine-tuning. While monolingual models remain competitive, this study highlights that multilingual models offer strong cross-lingual transfer capabilities in resource limited settings.