# FormosanBench: Benchmarking Low-Resource Austronesian Languages in the Era of Large Language Models

arXiv: https://arxiv.org/abs/2506.21563 · HTML: https://mlsanity.com/paper/2506.21563/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [Generating Wikipedia by Summarizing Long Sequences](https://mlsanity.com/paper/1801.10198/) (2018)
- [Massively Multilingual Neural Machine Translation](https://mlsanity.com/paper/1903.00089/) (2019)
- [A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios](https://mlsanity.com/paper/2010.12309/) (2020)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [IrokoBench: A New Benchmark for African Languages in the Age of Large Language Models](https://mlsanity.com/paper/2406.03368/) (2024)
- [Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold](https://mlsanity.com/paper/2206.09755/) (2022)
- [Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus](https://mlsanity.com/paper/2410.14815/) (2024)
- [Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control](https://mlsanity.com/paper/2207.02263/) (2022)
- [Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan Arabic Dialect](https://mlsanity.com/paper/2409.17912/) (2024)
- [Voices Unheard: NLP Resources and Models for Yor`ub'a Regional Dialects](https://mlsanity.com/paper/2406.19564/) (2024)
- [WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphs](https://mlsanity.com/paper/2209.13101/) (2022)
- [Enhancing Plagiarism Detection in Marathi with a Weighted Ensemble of TF-IDF and BERT Embeddings for Low-Resource Language Processing](https://mlsanity.com/paper/2501.05260/) (2025)

## Abstract (arXiv)

While large language models (LLMs) have demonstrated impressive performance across a wide range of natural language processing (NLP) tasks in high-resource languages, their capabilities in low-resource and minority languages remain significantly underexplored. Formosan languages -- a subgroup of Austronesian languages spoken in Taiwan -- are both linguistically rich and endangered, largely due to the sociolinguistic dominance of Mandarin. In this work, we introduce FORMOSANBENCH, the first benchmark for evaluating LLMs on low-resource Austronesian languages. It covers three endangered Formosan languages: Atayal, Amis, and Paiwan, across three core NLP tasks: machine translation, automatic speech recognition (ASR), and text summarization. We assess model performance in zero-shot, 10-shot, and fine-tuned settings using FORMOSANBENCH. Our results reveal a substantial performance gap between high-resource and Formosan languages. Existing LLMs consistently underperform across all tasks, with 10-shot learning and fine-tuning offering only limited improvements. These findings underscore the urgent need for more inclusive NLP technologies that can effectively support endangered and underrepresented languages. We release our datasets and code to facilitate future research in this direction.