# Improving Language Identification for Code-Switched Utterances with Integer Linear Programming

arXiv: https://arxiv.org/abs/2609.05099 · HTML: https://mlsanity.com/paper/2609.05099/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 52 references that are in the map.
- First posted on arXiv 4 Sep 2026.

## Builds on (most cited first, 24 of 24)

- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [MADLAD-400: A Multilingual And Document-Level Large Audited Dataset](https://mlsanity.com/paper/2309.04662/) (2023)
- [FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language](https://mlsanity.com/paper/2506.20920/) (2025)
- [Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages](https://mlsanity.com/paper/2305.12182/) (2023)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus](https://mlsanity.com/paper/2010.14571/) (2020)
- [A Survey of Code-switching: Linguistic and Social Perspectives for Language Technologies](https://mlsanity.com/paper/2301.01967/) (2023)
- [An Open Dataset and Model for Language Identification](https://mlsanity.com/paper/2305.13820/) (2023)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [Automatic Language Identification in Texts: A Survey](https://mlsanity.com/paper/1804.08186/) (2018)
- [AfroLID: A Neural Language Identification Tool for African Languages](https://mlsanity.com/paper/2210.11744/) (2022)
- [GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages](https://mlsanity.com/paper/2410.23825/) (2024)
- [GlotScript: A Resource and Tool for Low Resource Writing System Identification](https://mlsanity.com/paper/2309.13320/) (2023)
- [Subword-Level Language Identification for Intra-Word Code-Switching](https://mlsanity.com/paper/1904.01989/) (2019)
- [A Fast, Compact, Accurate Model for Language Identification of Codemixed Text](https://mlsanity.com/paper/1810.04142/) (2018)
- [MaskLID: Code-Switching Language Identification through Iterative Masking](https://mlsanity.com/paper/2406.06263/) (2024)
- [LanideNN: Multilingual Language Identification on Character Window](https://mlsanity.com/paper/1701.03338/) (2017)
- [Kallaama: A Transcribed Speech Dataset about Agriculture in the Three Most Widely Spoken Languages in Senegal](https://mlsanity.com/paper/2404.01991/) (2024)
- [CommonLID: Re-evaluating State-of-the-Art Language Identification Performance on Web Data](https://mlsanity.com/paper/2601.18026/) (2026)
- [The Integer Linear Programming Inference Cookbook](https://mlsanity.com/paper/2307.00171/) (2023)
- [Multi-label Scandinavian Language Identification (SLIDE)](https://mlsanity.com/paper/2502.06692/) (2025)

## Abstract (arXiv)

Automatic identification of code-switched (CS) utterances remains a challenge for language identification (LID) systems, causing such texts to be underrepresented in the training data of Large Language Models. In this paper, we revisit MaskLID, a state-of-the art approach for CS identification, which requires no training and detects arbitrary language combinations. We make three main contributions: (a) we reveal, and address, a major issue of MaskLID: its overreliance on word-level language association scores; (b) we reformulate the underlying optimization algorithm as an Integer Linear Program, enabling us to experiment with a large set of clear and interpretable constraints; (c) each of these improvements vastly improves the baseline system, as we illustrate in experiments involving 10~diverse languages, where we observe a strong boost in performance on CS benchmarks. We release our code and data for reproducibility.