# Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya

arXiv: https://arxiv.org/abs/2006.07698 · HTML: https://mlsanity.com/paper/2006.07698/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jun 2020.

## Built on by (most cited first, 8 of 8)

- [Evaluating the Capabilities of Large Language Models for Multi-label Emotion Understanding](https://mlsanity.com/paper/2412.17837/) (2024)
- [EthioLLM: Multilingual Large Language Models for Ethiopian Languages with Task Evaluation](https://mlsanity.com/paper/2403.13737/) (2024)
- [TIGQA:An Expert Annotated Question Answering Dataset in Tigrinya](https://mlsanity.com/paper/2404.17194/) (2024)
- [On the Universality of Deep Contextual Language Models](https://mlsanity.com/paper/2109.07140/) (2021)
- [Natural Language Processing for Tigrinya: Current State and Future Directions](https://mlsanity.com/paper/2507.17974/) (2025)
- [A Multi-Task Benchmark for Abusive Language Detection in Low-Resource Settings](https://mlsanity.com/paper/2505.12116/) (2025)
- [Analysing the Greek Parliament Records with Emotion Classification](https://mlsanity.com/paper/2205.12012/) (2022)
- [Exploring Cross-Lingual Knowledge Transfer via Transliteration-Based MLM Fine-Tuning for Critically Low-resource Chakma Language](https://mlsanity.com/paper/2510.09032/) (2025)

## Abstract (arXiv)

In recent years, transformer models have achieved great success in natural language processing (NLP) tasks. Most of the current state-of-the-art NLP results are achieved by using monolingual transformer models, where the model is pre-trained using a single language unlabelled text corpus. Then, the model is fine-tuned to the specific downstream task. However, the cost of pre-training a new transformer model is high for most languages. In this work, we propose a cost-effective transfer learning method to adopt a strong source language model, trained from a large monolingual corpus to a low-resource language. Thus, using XLNet language model, we demonstrate competitive performance with mBERT and a pre-trained target language model on the cross-lingual sentiment (CLS) dataset and on a new sentiment analysis dataset for low-resourced language Tigrinya. With only 10k examples of the given Tigrinya sentiment analysis dataset, English XLNet has achieved 78.88% F1-Score outperforming BERT and mBERT by 10% and 7%, respectively. More interestingly, fine-tuning (English) XLNet model on the CLS dataset has promising results compared to mBERT and even outperformed mBERT for one dataset of the Japanese language.