# FilBench: Can LLMs Understand and Generate Filipino? arXiv: https://arxiv.org/abs/2508.03523 ยท HTML: https://mlsanity.com/paper/2508.03523/ ## Key facts - Cited by 3 papers in the map (data through 14 Oct 2026). - Builds on 34 of its 72 references that are in the map. - First posted on arXiv 5 Aug 2025. ## Builds on (most cited first, 30 of 34) - [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023) - [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019) - [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023) - [Sentiment Analysis in the Era of Large Language Models: A Reality Check](https://mlsanity.com/paper/2305.15005/) (2023) - [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023) - [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023) - [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024) - [GPT-NER: Named Entity Recognition via Large Language Models](https://mlsanity.com/paper/2304.10428/) (2023) - [SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects](https://mlsanity.com/paper/2309.07445/) (2023) - [The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT](https://mlsanity.com/paper/2010.06354/) (2020) - [OLMES: A Standard for Language Model Evaluations](https://mlsanity.com/paper/2406.08446/) (2024) - [TICO-19: the Translation Initiative for Covid-19](https://mlsanity.com/paper/2007.01788/) (2020) - [Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages](https://mlsanity.com/paper/2410.16153/) (2024) - [Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM](https://mlsanity.com/paper/2303.01911/) (2023) - [Towards a Common Understanding of Contributing Factors for Cross-Lingual Transfer in Multilingual Language Models: A Review](https://mlsanity.com/paper/2305.16768/) (2023) - [Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning](https://mlsanity.com/paper/2310.13448/) (2023) - [Evaluating Cultural and Social Awareness of LLM Web Agents](https://mlsanity.com/paper/2410.23252/) (2024) - [SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages](https://mlsanity.com/paper/2406.10118/) (2024) - [SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages](https://mlsanity.com/paper/2407.19672/) (2024) - [Zero-shot cross-lingual transfer language selection using linguistic similarity](https://mlsanity.com/paper/2301.13720/) (2023) - [Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark](https://mlsanity.com/paper/2311.09122/) (2023) - [Improving Large-scale Language Models and Resources for Filipino](https://mlsanity.com/paper/2111.06053/) (2021) - [SEA-HELM: Southeast Asian Holistic Evaluation of Language Models](https://mlsanity.com/paper/2502.14301/) (2025) - [Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers](https://mlsanity.com/paper/2503.00865/) (2025) - [Establishing Baselines for Text Classification in Low-Resource Languages](https://mlsanity.com/paper/2005.02068/) (2020) - [SeaExam and SeaBench: Benchmarking LLMs with Local Multilingual Questions in Southeast Asia](https://mlsanity.com/paper/2502.06298/) (2025) - [BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanism](https://mlsanity.com/paper/2412.17933/) (2024) - [Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasets](https://mlsanity.com/paper/2010.11574/) (2020) - [A Baseline Readability Model for Cebuano](https://mlsanity.com/paper/2203.17225/) (2022) - [Automatic Readability Assessment for Closely Related Languages](https://mlsanity.com/paper/2305.13478/) (2023) ## Built on by (most cited first, 3 of 3) - [Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages](https://mlsanity.com/paper/2509.02160/) (2025) - [KatotohananQA: Evaluating Truthfulness of Large Language Models in Filipino](https://mlsanity.com/paper/2509.06065/) (2025) - [Forgotten Words: Benchmarking NeoBERT for Dementia Detection in Low-Resource Conversational Filipino and English Speech](https://mlsanity.com/paper/2605.26007/) (2026) ## Abstract (arXiv) Despite the impressive performance of LLMs on English-based tasks, little is known about their capabilities in specific languages such as Filipino. In this work, we address this gap by introducing FilBench, a Filipino-centric benchmark designed to evaluate LLMs across a diverse set of tasks and capabilities in Filipino, Tagalog, and Cebuano. We carefully curate the tasks in FilBench to reflect the priorities and trends of NLP research in the Philippines such as Cultural Knowledge, Classical NLP, Reading Comprehension, and Generation. By evaluating 27 state-of-the-art LLMs on FilBench, we find that several LLMs suffer from reading comprehension and translation capabilities. Our results indicate that FilBench is challenging, with the best model, GPT-4o, achieving only a score of 72.23%. Moreover, we also find that models trained specifically for Southeast Asian languages tend to underperform on FilBench, with the highest-performing model, SEA-LION v3 70B, achieving only a score of 61.07%. Our work demonstrates the value of curating language-specific LLM benchmarks to aid in driving progress on Filipino NLP and increasing the inclusion of Philippine languages in LLM development.