# Sequence-to-Sequence Lexical Normalization with Multilingual Transformers

arXiv: https://arxiv.org/abs/2110.02869 · HTML: https://mlsanity.com/paper/2110.02869/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 58 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 16 of 16)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://mlsanity.com/paper/2008.00401/) (2020)
- [Predicting the Type and Target of Offensive Posts in Social Media](https://mlsanity.com/paper/1902.09666/) (2019)
- [SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)](https://mlsanity.com/paper/1903.08983/) (2019)
- [SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020)](https://mlsanity.com/paper/2006.07235/) (2020)
- [Depression and Self-Harm Risk Assessment in Online Forums](https://mlsanity.com/paper/1709.01848/) (2017)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [Latent Suicide Risk Detection on Microblog via Suicide-Oriented Word Embeddings and Layered Attention](https://mlsanity.com/paper/1910.12038/) (2019)
- [DaN+: Danish Nested Named Entities and Lexical Normalization](https://mlsanity.com/paper/2105.11301/) (2021)
- [Early Risk Detection of Pathological Gambling, Self-Harm and Depression Using BERT](https://mlsanity.com/paper/2106.16175/) (2021)
- [An Exploratory Analysis of the Relation Between Offensive Language and Mental Health](https://mlsanity.com/paper/2105.14888/) (2021)
- [MoNoise: Modeling Noise Using a Modular Normalization System](https://mlsanity.com/paper/1710.03476/) (2017)
- [Attention-Based LSTM for Psychological Stress Detection from Spoken Language Using Distant Supervision](https://mlsanity.com/paper/1805.12307/) (2018)
- [Adapting Sequence to Sequence models for Text Normalization in Social Media](https://mlsanity.com/paper/1904.06100/) (2019)
- [Lexical Normalization for Code-switched Data and its Effect on POS-tagging](https://mlsanity.com/paper/2006.01175/) (2020)
- [Normalization of Transliterated Words in Code-Mixed Data Using Seq2Seq Model & Levenshtein Distance](https://mlsanity.com/paper/1805.08701/) (2018)

## Built on by (most cited first, 7 of 7)

- ['UFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5](https://mlsanity.com/paper/2110.15248/) (2021)
- [BLUE at Memotion 2.0 2022: You have my Image, my Text and my Transformer](https://mlsanity.com/paper/2202.07543/) (2022)
- [Script Normalization for Unconventional Writing of Under-Resourced Languages in Bilingual Communities](https://mlsanity.com/paper/2305.16407/) (2023)
- [A Weakly Supervised Data Labeling Framework for Machine Lexical Normalization in Vietnamese Social Media](https://mlsanity.com/paper/2409.20467/) (2024)
- [Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages](https://mlsanity.com/paper/2505.22273/) (2025)
- [MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages](https://mlsanity.com/paper/2601.16623/) (2026)
- [Historical German Text Normalization Using Type- and Token-Based Language Modeling](https://mlsanity.com/paper/2409.02841/) (2024)

## Abstract (arXiv)

Current benchmark tasks for natural language processing contain text that is qualitatively different from the text used in informal day to day digital communication. This discrepancy has led to severe performance degradation of state-of-the-art NLP models when fine-tuned on real-world data. One way to resolve this issue is through lexical normalization, which is the process of transforming non-standard text, usually from social media, into a more standardized form. In this work, we propose a sentence-level sequence-to-sequence model based on mBART, which frames the problem as a machine translation problem. As the noisy text is a pervasive problem across languages, not just English, we leverage the multi-lingual pre-training of mBART to fine-tune it to our data. While current approaches mainly operate at the word or subword level, we argue that this approach is straightforward from a technical standpoint and builds upon existing pre-trained transformer networks. Our results show that while word-level, intrinsic, performance evaluation is behind other methods, our model improves performance on extrinsic, downstream tasks through normalization compared to models operating on raw, unprocessed, social media text.