mlsanityData to 14 Oct 2026

Improving Large-scale Language Models and Resources for Filipino

arXiv 2111.06053 ↗First posted 11 Nov 2021cs.CLcited by 20

Not on the map. Why?

Abstract (arXiv)

In this paper, we improve on existing language resources for the low-resource Filipino language in two ways. First, we outline the construction of the TLUnified dataset, a large-scale pretraining corpus that serves as an improvement over smaller existing pretraining datasets for the language in terms of scale and topic variety. Second, we pretrain new Transformer language models following the RoBERTa pretraining technique to supplant existing models trained with small corpora. Our new RoBERTa models show significant improvements over existing Filipino models in three benchmark datasets with an average gain of 4.47% test accuracy across the three classification tasks of varying difficulty.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 11 of its 21 references that are in the map. First posted on arXiv 11 Nov 2021.

Builds on

11 of 21 references.

References · 11

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  6. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
  7. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  8. 2021Do Syntactic Probes Probe Syntax? Experiments with Jabberwocky Probingcited by 28
  9. 2020Establishing Baselines for Text Classification in Low-Resource Languagescited by 14
  10. 2019When Low Resource NLP Meets Unsupervised Language Model: Meta-pretraining Then Meta-learning for Few-shot Text Classificationcited by 13
  11. 2020Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasetscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 1; 2025: 7; 2026: 4.

20 papers.

Sort:

By area:

  1. 2025Crowdsource, Crawl, or Generate? Creating SEA-VL, a Multicultural Vision-Language Dataset for Southeast Asiacited by 9
  2. 2023Developing a Named Entity Recognition Dataset for Tagalogcited by 7
  3. 2022MTet: Multi-domain Translation for English and Vietnamesecited by 5
  4. 2025Batayan: A Filipino NLP benchmark for evaluating Large Language Modelscited by 5
  5. 2022Towards Automatic Construction of Filipino WordNet: Word Sense Induction and Synset Induction Using Sentence Embeddingscited by 3
  6. 2023calamanCy: A Tagalog Natural Language Processing Toolkitcited by 3
  7. 2025FilBench: Can LLMs Understand and Generate Filipino?cited by 3
  8. 2022Using Synthetic Data for Conversational Response Generation in Low-resource Settingscited by 2
  9. 2025Bias Attribution in Filipino Language Models: Extending a Bias Interpretability Metric for Application on Agglutinative Languagescited by 2
  10. 2024MasonTigers at SemEval-2024 Task 1: An Ensemble Approach for Semantic Textual Relatednesscited by 1
  11. 2025Extracting General-use Transformers for Low-resource Languages via Knowledge Distillationcited by 1
  12. 2026Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Modelscited by 1
8 more
  1. 2022Look Ma, Only 400 Samples! Revisiting the Effectiveness of Automatic N-Gram Rule Generation for Spelling Normalization in Filipinocited by 0
  2. 2023Label-Aware Automatic Verbalizer for Few-Shot Text Classificationcited by 0
  3. 2023Samsung R&D Institute Philippines at WMT 2023cited by 0
  4. 2025FiLLM -- A Filipino-optimized Large Language Model based on Southeast Asia Large Language Model (SEALLM)cited by 0
  5. 2025The UD-NewsCrawl Treebank: Reflections and Challenges from a Large-scale Tagalog Syntactic Annotation Projectcited by 0
  6. 2026Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speechcited by 0
  7. 2026PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipinocited by 0
  8. 2026TDR: Tagalog Diacritic Restoration Datasetcited by 0