# A Fast, Compact, Accurate Model for Language Identification of Codemixed Text

arXiv: https://arxiv.org/abs/1810.04142 · HTML: https://mlsanity.com/paper/1810.04142/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Oct 2018.

## Built on by (most cited first, 11 of 11)

- [Scaling Up Influence Functions](https://mlsanity.com/paper/2112.03052/) (2021)
- [Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability](https://mlsanity.com/paper/2305.10266/) (2023)
- [AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters](https://mlsanity.com/paper/2401.06408/) (2024)
- [Dialect-robust Evaluation of Generated Text](https://mlsanity.com/paper/2211.00922/) (2022)
- [Language Detection Engine for Multilingual Texting on Mobile Devices](https://mlsanity.com/paper/2101.03963/) (2021)
- [DAMP: Doubly Aligned Multilingual Parser for Task-Oriented Dialogue](https://mlsanity.com/paper/2212.08054/) (2022)
- [Computational Approaches to Arabic-English Code-Switching](https://mlsanity.com/paper/2410.13318/) (2024)
- [Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global South](https://mlsanity.com/paper/2605.19190/) (2026)
- [Automatic Language Identification for Celtic Texts](https://mlsanity.com/paper/2203.04831/) (2022)
- [Detecting Latin in Historical Books with Large Language Models: A Multimodal Benchmark](https://mlsanity.com/paper/2510.19585/) (2025)
- [Improving Language Identification for Code-Switched Utterances with Integer Linear Programming](https://mlsanity.com/paper/2609.05099/) (2026)

## Abstract (arXiv)

We address fine-grained multilingual language identification: providing a language code for every token in a sentence, including codemixed text containing multiple languages. Such text is prevalent online, in documents, social media, and message boards. We show that a feed-forward network with a simple globally constrained decoder can accurately and rapidly label both codemixed and monolingual text in 100 languages and 100 language pairs. This model outperforms previously published multilingual approaches in terms of both accuracy and speed, yielding an 800x speed-up and a 19.5% averaged absolute gain on three codemixed datasets. It furthermore outperforms several benchmark systems on monolingual language identification.