# Establishing Baselines for Text Classification in Low-Resource Languages

arXiv: https://arxiv.org/abs/2005.02068 · HTML: https://mlsanity.com/paper/2005.02068/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2020.

## Built on by (most cited first, 14 of 14)

- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation](https://mlsanity.com/paper/2104.08200/) (2021)
- [Goldfish: Monolingual Language Models for 350 Languages](https://mlsanity.com/paper/2408.10441/) (2024)
- [GreenPLM: Cross-Lingual Transfer of Monolingual Pre-Trained Language Models at Almost No Cost](https://mlsanity.com/paper/2211.06993/) (2022)
- [Improving Large-scale Language Models and Resources for Filipino](https://mlsanity.com/paper/2111.06053/) (2021)
- [A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models](https://mlsanity.com/paper/2201.05601/) (2022)
- [Cracking Double-Blind Review: Authorship Attribution with Deep Learning](https://mlsanity.com/paper/2211.07467/) (2022)
- [Text classification dataset and analysis for Uzbek language](https://mlsanity.com/paper/2302.14494/) (2023)
- [Benchmarking Multilabel Topic Classification in the Kyrgyz Language](https://mlsanity.com/paper/2308.15952/) (2023)
- [FilBench: Can LLMs Understand and Generate Filipino?](https://mlsanity.com/paper/2508.03523/) (2025)
- [L3Cube-MahaEmotions: A Marathi Emotion Recognition Dataset with Synthetic Annotations using CoTR prompting and Large Language Models](https://mlsanity.com/paper/2506.00863/) (2025)
- [Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters](https://mlsanity.com/paper/2111.06971/) (2021)
- [Lutma: a Frame-Making Tool for Collaborative FrameNet Development](https://mlsanity.com/paper/2205.11840/) (2022)
- [FiLLM -- A Filipino-optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)](https://mlsanity.com/paper/2505.18995/) (2025)

## Abstract (arXiv)

While transformer-based finetuning techniques have proven effective in tasks that involve low-resource, low-data environments, a lack of properly established baselines and benchmark datasets make it hard to compare different approaches that are aimed at tackling the low-resource setting. In this work, we provide three contributions. First, we introduce two previously unreleased datasets as benchmark datasets for text classification and low-resource multilabel text classification for the low-resource language Filipino. Second, we pretrain better BERT and DistilBERT models for use within the Filipino setting. Third, we introduce a simple degradation test that benchmarks a model's resistance to performance degradation as the number of training samples are reduced. We analyze our pretrained model's degradation speeds and look towards the use of this method for comparing models aimed at operating within the low-resource setting. We release all our models and datasets for the research community to use.