# Automatic Standardization of Colloquial Persian

arXiv: https://arxiv.org/abs/2012.05879 · HTML: https://mlsanity.com/paper/2012.05879/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2020.

## Built on by (most cited first, 4 of 4)

- [Developing an Informal-Formal Persian Corpus](https://mlsanity.com/paper/2308.05336/) (2023)
- [PERCORE: A Deep Learning-Based Framework for Persian Spelling Correction with Phonetic Analysis](https://mlsanity.com/paper/2407.14789/) (2024)
- [Persian Slang Text Conversion to Formal and Deep Learning of Persian Short Texts on Social Media for Sentiment Classification](https://mlsanity.com/paper/2403.06023/) (2024)
- [Improving the quality of Persian clinical text with a novel spelling correction system](https://mlsanity.com/paper/2408.03622/) (2024)

## Abstract (arXiv)

The Iranian Persian language has two varieties: standard and colloquial. Most natural language processing tools for Persian assume that the text is in standard form: this assumption is wrong in many real applications especially web content. This paper describes a simple and effective standardization approach based on sequence-to-sequence translation. We design an algorithm for generating artificial parallel colloquial-to-standard data for learning a sequence-to-sequence model. Moreover, we annotate a publicly available evaluation data consisting of 1912 sentences from a diverse set of domains. Our intrinsic evaluation shows a higher BLEU score of 62.8 versus 61.7 compared to an off-the-shelf rule-based standardization model in which the original text has a BLEU score of 46.4. We also show that our model improves English-to-Persian machine translation in scenarios for which the training data is from colloquial Persian with 1.4 absolute BLEU score difference in the development data, and 0.8 in the test data.