# Language Identification for Austronesian Languages

arXiv: https://arxiv.org/abs/2206.04327 · HTML: https://mlsanity.com/paper/2206.04327/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 32 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 8 of 8)

- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [FastText.zip: Compressing text classification models](https://mlsanity.com/paper/1612.03651/) (2016)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [Automatic Language Identification in Texts: A Survey](https://mlsanity.com/paper/1804.08186/) (2018)
- [Overview for the Second Shared Task on Language Identification in Code-Switched Data](https://mlsanity.com/paper/1909.13016/) (2019)
- [LanideNN: Multilingual Language Identification on Character Window](https://mlsanity.com/paper/1701.03338/) (2017)
- [Analyzing the Surprising Variability in Word Embedding Stability Across Languages](https://mlsanity.com/paper/2004.14876/) (2020)
- [Hierarchical Character-Word Models for Language Identification](https://mlsanity.com/paper/1608.03030/) (2016)

## Built on by (most cited first, 3 of 3)

- [Geographically-Informed Language Identification](https://mlsanity.com/paper/2403.09892/) (2024)
- [Comparing Measures of Linguistic Diversity Across Social Media Language Data and Census Data at Subnational Geographic Areas](https://mlsanity.com/paper/2308.10452/) (2023)
- [Syntactic Variation Across the Grammar: Modelling a Complex Adaptive System](https://mlsanity.com/paper/2309.11869/) (2023)

## Abstract (arXiv)

This paper provides language identification models for low- and under-resourced languages in the Pacific region with a focus on previously unavailable Austronesian languages. Accurate language identification is an important part of developing language resources. The approach taken in this paper combines 29 Austronesian languages with 171 non-Austronesian languages to create an evaluation set drawn from eight data sources. After evaluating six approaches to language identification, we find that a classifier based on skip-gram embeddings reaches a significantly higher performance than alternate methods. We then systematically increase the number of non-Austronesian languages in the model up to a total of 800 languages to evaluate whether an increased language inventory leads to less precise predictions for the Austronesian languages of interest. This evaluation finds that there is only a minimal impact on accuracy caused by increasing the inventory of non-Austronesian languages. Further experiments adapt these language identification models for code-switching detection, achieving high accuracy across all 29 languages.