# PALI: A Language Identification Benchmark for Perso-Arabic Scripts

arXiv: https://arxiv.org/abs/2304.01322 · HTML: https://mlsanity.com/paper/2304.01322/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 33 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 7 of 7)

- [NADI 2020: The First Nuanced Arabic Dialect Identification Shared Task](https://mlsanity.com/paper/2010.11334/) (2020)
- [Automatic Language Identification in Texts: A Survey](https://mlsanity.com/paper/1804.08186/) (2018)
- [A Rule-based Kurdish Text Transliteration System](https://mlsanity.com/paper/1811.10278/) (2018)
- [Discriminating Between Similar Nordic Languages](https://mlsanity.com/paper/2012.06431/) (2020)
- [Graphemic Normalization of the Perso-Arabic Script](https://mlsanity.com/paper/2210.12273/) (2022)
- [LIDE: Language Identification from Text Documents](https://mlsanity.com/paper/1701.03682/) (2017)
- [Discriminating between Indo-Aryan Languages Using SVM Ensembles](https://mlsanity.com/paper/1807.03108/) (2018)

## Built on by (most cited first, 7 of 7)

- [GlotLID: Language Identification for Low-Resource Languages](https://mlsanity.com/paper/2310.16248/) (2023)
- [NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations](https://mlsanity.com/paper/2312.06352/) (2023)
- [Advancing Post-OCR Correction: A Comparative Study of Synthetic Data](https://mlsanity.com/paper/2408.02253/) (2024)
- [Language and Speech Technology for Central Kurdish Varieties](https://mlsanity.com/paper/2403.01983/) (2024)
- [Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities](https://mlsanity.com/paper/2305.16407/) (2023)
- [Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models](https://mlsanity.com/paper/2606.05531/) (2026)
- [Script-Agnostic Language Identification](https://mlsanity.com/paper/2406.17901/) (2024)

## Abstract (arXiv)

The Perso-Arabic scripts are a family of scripts that are widely adopted and used by various linguistic communities around the globe. Identifying various languages using such scripts is crucial to language technologies and challenging in low-resource setups. As such, this paper sheds light on the challenges of detecting languages using Perso-Arabic scripts, especially in bilingual communities where ``unconventional'' writing is practiced. To address this, we use a set of supervised techniques to classify sentences into their languages. Building on these, we also propose a hierarchical model that targets clusters of languages that are more often confused by the classifiers. Our experiment results indicate the effectiveness of our solutions.