mlsanityData to 14 Oct 2026

FilBench: Can LLMs Understand and Generate Filipino?

arXiv 2508.03523 ↗First posted 5 Aug 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Despite the impressive performance of LLMs on English-based tasks, little is known about their capabilities in specific languages such as Filipino. In this work, we address this gap by introducing FilBench, a Filipino-centric benchmark designed to evaluate LLMs across a diverse set of tasks and capabilities in Filipino, Tagalog, and Cebuano. We carefully curate the tasks in FilBench to reflect the priorities and trends of NLP research in the Philippines such as Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation. By evaluating 27 state-of-the-art LLMs on FilBench, we find that several LLMs suffer from reading comprehension and translation capabilities. Our results indicate that FilBench is challenging, with the best model, GPT-4o, achieving only a score of 72.23%. Moreover, we also find that models trained specifically for Southeast Asian languages tend to underperform on FilBench, with the highest-performing model, SEA-LION v3 70B, achieving only a score of 61.07%. Our work demonstrates the value of curating language-specific LLM benchmarks to aid in driving progress on Filipino NLP and increasing the inclusion of Philippine languages in LLM development.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 34 of its 72 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

34 of 72 references.

References · 34

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  3. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  4. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  5. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 250
  6. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 206
  7. 2024Tower: An Open Multilingual Large Language Model for Translation-Related Taskscited by 206
  8. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 160
  9. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  10. 2020The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MTcited by 128
  11. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  12. 2020TICO-19: the Translation Initiative for Covid-19cited by 68
22 more
  1. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 64
  2. 2023Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOMcited by 59
  3. 2023Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Reviewcited by 50
  4. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 49
  5. 2024Evaluating Cultural and Social Awareness of LLM Web Agentscited by 45
  6. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  7. 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 36
  8. 2023Zero-shot cross-lingual transfer language selection using linguistic similaritycited by 28
  9. 2023Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmarkcited by 26
  10. 2021Improving Large-scale Language Models and Resources for Filipinocited by 20
  11. 2025SEA-HELM: Southeast Asian Holistic Evaluation of Language Modelscited by 19
  12. 2025Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakerscited by 16
  13. 2020Establishing Baselines for Text Classification in Low-Resource Languagescited by 14
  14. 2025SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asiacited by 13
  15. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  16. 2020Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasetscited by 10
  17. 2022A Baseline Readability Model for Cebuanocited by 10
  18. 2023Automatic Readability Assessment for Closely Related Languagescited by 9
  19. 2025Lugha-Llama: Adapting Large Language Models for African Languagescited by 8
  20. 2023Developing a Named Entity Recognition Dataset for Tagalogcited by 7
  21. 2024Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sensecited by 4
  22. 2023CebuaNER: A New Baseline Cebuano Named Entity Recognition Modelcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languagescited by 3
  2. 2025KatotohananQA: Evaluating Truthfulness of Large Language Models in Filipinocited by 0
  3. 2026Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speechcited by 0