# Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities

arXiv: https://arxiv.org/abs/2305.16407 · HTML: https://mlsanity.com/paper/2305.16407/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 73 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 18 of 18)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Joey NMT: A Minimalist NMT Toolkit for Novices](https://mlsanity.com/paper/1907.12484/) (2019)
- [RNN Approaches to Text Normalization: A Challenge](https://mlsanity.com/paper/1611.00068/) (2016)
- [Domain, Translationese and Noise in Synthetic Data for Neural Machine Translation](https://mlsanity.com/paper/1911.03362/) (2019)
- [A Rule-based Kurdish Text Transliteration System](https://mlsanity.com/paper/1811.10278/) (2018)
- [MoNoise: Modeling Noise Using a Modular Normalization System](https://mlsanity.com/paper/1710.03476/) (2017)
- [Sequence-to-Sequence Lexical Normalization with Multilingual Transformers](https://mlsanity.com/paper/2110.02869/) (2021)
- [PALI: A Language Identification Benchmark for Perso-Arabic Scripts](https://mlsanity.com/paper/2304.01322/) (2023)
- [Towards Machine Translation for the Kurdish Language](https://mlsanity.com/paper/2010.06041/) (2020)
- [Graphemic Normalization of the Perso-Arabic Script](https://mlsanity.com/paper/2210.12273/) (2022)
- [Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance](https://mlsanity.com/paper/1805.08701/) (2018)
- [Towards robust word embeddings for noisy texts](https://mlsanity.com/paper/1911.10876/) (2019)
- [Joint Persian Word Segmentation Correction and Zero-Width Non-Joiner Recognition Using BERT](https://mlsanity.com/paper/2010.00287/) (2020)
- [Text normalization for low-resource languages: the case of Ligurian](https://mlsanity.com/paper/2206.07861/) (2022)
- [Beyond Arabic: Software for Perso-Arabic Script Manipulation](https://mlsanity.com/paper/2301.11406/) (2023)

## Built on by (most cited first, 2 of 2)

- [Language and Speech Technology for Central Kurdish Varieties](https://mlsanity.com/paper/2403.01983/) (2024)
- [textitVersteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language](https://mlsanity.com/paper/2603.28213/) (2026)

## Abstract (arXiv)

The wide accessibility of social media has provided linguistically under-represented communities with an extraordinary opportunity to create content in their native languages. This, however, comes with certain challenges in script normalization, particularly where the speakers of a language in a bilingual community rely on another script or orthography to write their native language. This paper addresses the problem of script normalization for several such languages that are mainly written in a Perso-Arabic script. Using synthetic data with various levels of noise and a transformer-based model, we demonstrate that the problem can be effectively remediated. We conduct a small-scale evaluation of real data as well. Our experiments indicate that script normalization is also beneficial to improve the performance of downstream tasks such as machine translation and language identification.