# Evaluating Language Model Finetuning Techniques for Low-resource Languages

arXiv: https://arxiv.org/abs/1907.00409 · HTML: https://mlsanity.com/paper/1907.00409/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Jun 2019.

## Built on by (most cited first, 12 of 12)

- [State-of-the-art generalisation research in NLP: A taxonomy and review](https://mlsanity.com/paper/2210.03050/) (2022)
- [BERT Embeddings for Automatic Readability Assessment](https://mlsanity.com/paper/2106.07935/) (2021)
- [Prompt-and-Align: Prompt-Based Social Alignment for Few-Shot Fake News Detection](https://mlsanity.com/paper/2309.16424/) (2023)
- [Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning](https://mlsanity.com/paper/2410.08598/) (2024)
- [Sharing to learn and learning to share; Fitting together Meta-Learning, Multi-Task Learning, and Transfer Learning: A meta review](https://mlsanity.com/paper/2111.12146/) (2021)
- [Batayan: A Filipino NLP benchmark for evaluating Large Language Models](https://mlsanity.com/paper/2502.14911/) (2025)
- [Monolingual and Multilingual Misinformation Detection for Low-Resource Languages: A Comprehensive Survey](https://mlsanity.com/paper/2410.18390/) (2024)
- [SEA-BED: How Do Embedding Models Represent Southeast Asian Languages?](https://mlsanity.com/paper/2508.12243/) (2025)
- [Ensuring the Inclusive Use of Natural Language Processing in the Global Response to COVID-19](https://mlsanity.com/paper/2108.10791/) (2021)
- [A Unified Training Process for Fake News Detection based on Fine-Tuned BERT Model](https://mlsanity.com/paper/2202.01907/) (2022)
- [Cross-lingual Transfer Learning for Fake News Detector in a Low-Resource Language](https://mlsanity.com/paper/2208.12482/) (2022)
- [Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks](https://mlsanity.com/paper/2403.15458/) (2024)

## Abstract (arXiv)

Unlike mainstream languages (such as English and French), low-resource languages often suffer from a lack of expert-annotated corpora and benchmark resources that make it hard to apply state-of-the-art techniques directly. In this paper, we alleviate this scarcity problem for the low-resourced Filipino language in two ways. First, we introduce a new benchmark language modeling dataset in Filipino which we call WikiText-TL-39. Second, we show that language model finetuning techniques such as BERT and ULMFiT can be used to consistently train robust classifiers in low-resource settings, experiencing at most a 0.0782 increase in validation error when the number of training examples is decreased from 10K to 1K while finetuning using a privately-held sentiment dataset.