# Lexical Normalization for Code-switched Data and its Effect on POS-tagging

arXiv: https://arxiv.org/abs/2006.01175 · HTML: https://mlsanity.com/paper/2006.01175/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Jun 2020.

## Built on by (most cited first, 7 of 7)

- [Sequence-to-Sequence Lexical Normalization with Multilingual Transformers](https://mlsanity.com/paper/2110.02869/) (2021)
- ['UFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5](https://mlsanity.com/paper/2110.15248/) (2021)
- [Resources for Turkish Natural Language Processing: A critical survey](https://mlsanity.com/paper/2204.05042/) (2022)
- [Neural Text Normalization for Luxembourgish using Real-Life Variation Data](https://mlsanity.com/paper/2412.09383/) (2024)
- [MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages](https://mlsanity.com/paper/2601.16623/) (2026)
- [Variation is the Norm: Embracing Sociolinguistics in NLP](https://mlsanity.com/paper/2603.24222/) (2026)
- [Beyond Supervised Clarification: Input Rewriting with LLMs for Dialogue Discourse Parsing](https://mlsanity.com/paper/2607.01964/) (2026)

## Abstract (arXiv)

Lexical normalization, the translation of non-canonical data to standard language, has shown to improve the performance of manynatural language processing tasks on social media. Yet, using multiple languages in one utterance, also called code-switching (CS), is frequently overlooked by these normalization systems, despite its common use in social media. In this paper, we propose three normalization models specifically designed to handle code-switched data which we evaluate for two language pairs: Indonesian-English (Id-En) and Turkish-German (Tr-De). For the latter, we introduce novel normalization layers and their corresponding language ID and POS tags for the dataset, and evaluate the downstream effect of normalization on POS tagging. Results show that our CS-tailored normalization models outperform Id-En state of the art and Tr-De monolingual models, and lead to 5.4% relative performance increase for POS tagging as compared to unnormalized input.