# Improving Large-scale Language Models and Resources for Filipino

arXiv: https://arxiv.org/abs/2111.06053 · HTML: https://mlsanity.com/paper/2111.06053/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 11 Nov 2021.

## Builds on (most cited first, 11 of 11)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia](https://mlsanity.com/paper/1907.05791/) (2019)
- [CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairs](https://mlsanity.com/paper/1911.06154/) (2019)
- [Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probing](https://mlsanity.com/paper/2106.02559/) (2021)
- [Establishing Baselines for Text Classification in Low-Resource Languages](https://mlsanity.com/paper/2005.02068/) (2020)
- [When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classification](https://mlsanity.com/paper/1908.08788/) (2019)
- [Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasets](https://mlsanity.com/paper/2010.11574/) (2020)

## Built on by (most cited first, 20 of 20)

- [Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asia](https://mlsanity.com/paper/2503.07920/) (2025)
- [Developing a Named Entity Recognition Dataset for Tagalog](https://mlsanity.com/paper/2311.07161/) (2023)
- [MTet: Multi-domain Translation for English and Vietnamese](https://mlsanity.com/paper/2210.05610/) (2022)
- [Batayan: A Filipino NLP benchmark for evaluating Large Language Models](https://mlsanity.com/paper/2502.14911/) (2025)
- [Towards Automatic Construction of Filipino WordNet: Word Sense Induction and Synset Induction Using Sentence Embeddings](https://mlsanity.com/paper/2204.03251/) (2022)
- [calamanCy: A Tagalog Natural Language Processing Toolkit](https://mlsanity.com/paper/2311.07171/) (2023)
- [FilBench: Can LLMs Understand and Generate Filipino?](https://mlsanity.com/paper/2508.03523/) (2025)
- [Using Synthetic Data for Conversational Response Generation in Low-resource Settings](https://mlsanity.com/paper/2204.02653/) (2022)
- [Bias Attribution in Filipino Language Models: Extending a Bias Interpretability Metric for Application on Agglutinative Languages](https://mlsanity.com/paper/2506.07249/) (2025)
- [MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatedness](https://mlsanity.com/paper/2403.14990/) (2024)
- [Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation](https://mlsanity.com/paper/2501.12660/) (2025)
- [Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models](https://mlsanity.com/paper/2602.14466/) (2026)
- [Look Ma, Only 400 Samples! Revisiting the Effectiveness of Automatic N-Gram Rule Generation for Spelling Normalization in Filipino](https://mlsanity.com/paper/2210.02675/) (2022)
- [Label-Aware Automatic Verbalizer for Few-Shot Text Classification](https://mlsanity.com/paper/2310.12778/) (2023)
- [Samsung R&D Institute Philippines at WMT 2023](https://mlsanity.com/paper/2310.16322/) (2023)
- [FiLLM -- A Filipino-optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)](https://mlsanity.com/paper/2505.18995/) (2025)
- [The UD-NewsCrawl Treebank: Reflections and Challenges from a Large-scale Tagalog Syntactic Annotation Project](https://mlsanity.com/paper/2505.20428/) (2025)
- [Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech](https://mlsanity.com/paper/2605.26007/) (2026)
- [PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino](https://mlsanity.com/paper/2606.15144/) (2026)
- [TDR: Tagalog Diacritic Restoration Dataset](https://mlsanity.com/paper/2610.17597/) (2026)

## Abstract (arXiv)

In this paper, we improve on existing language resources for the low-resource Filipino language in two ways. First, we outline the construction of the TLUnified dataset, a large-scale pretraining corpus that serves as an improvement over smaller existing pretraining datasets for the language in terms of scale and topic variety. Second, we pretrain new Transformer language models following the RoBERTa pretraining technique to supplant existing models trained with small corpora. Our new RoBERTa models show significant improvements over existing Filipino models in three benchmark datasets with an average gain of 4.47% test accuracy across the three classification tasks of varying difficulty.