# Adapting Language Models to Indonesian Local Languages: An Empirical Study of Language Transferability on Zero-Shot Settings

arXiv: https://arxiv.org/abs/2507.01645 · HTML: https://mlsanity.com/paper/2507.01645/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://mlsanity.com/paper/2012.15562/) (2020)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting](https://mlsanity.com/paper/2212.09535/) (2022)
- [NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages](https://mlsanity.com/paper/2205.15960/) (2022)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Expanding Pretrained Models to Thousands More Languages via Lexicon-based Adaptation](https://mlsanity.com/paper/2203.09435/) (2022)
- [Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation](https://mlsanity.com/paper/2105.03953/) (2021)
- [KinyaBERT: a Morphology-aware Kinyarwanda Language Model](https://mlsanity.com/paper/2203.08459/) (2022)

## Abstract (arXiv)

In this paper, we investigate the transferability of pre-trained language models to low-resource Indonesian local languages through the task of sentiment analysis. We evaluate both zero-shot performance and adapter-based transfer on ten local languages using models of different types: a monolingual Indonesian BERT, multilingual models such as mBERT and XLM-R, and a modular adapter-based approach called MAD-X. To better understand model behavior, we group the target languages into three categories: seen (included during pre-training), partially seen (not included but linguistically related to seen languages), and unseen (absent and unrelated in pre-training data). Our results reveal clear performance disparities across these groups: multilingual models perform best on seen languages, moderately on partially seen ones, and poorly on unseen languages. We find that MAD-X significantly improves performance, especially for seen and partially seen languages, without requiring labeled data in the target language. Additionally, we conduct a further analysis on tokenization and show that while subword fragmentation and vocabulary overlap with Indonesian correlate weakly with prediction quality, they do not fully explain the observed performance. Instead, the most consistent predictor of transfer success is the model's prior exposure to the language, either directly or through a related language.