# 'UFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5

arXiv: https://arxiv.org/abs/2110.15248 · HTML: https://mlsanity.com/paper/2110.15248/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [GECToR -- Grammatical Error Correction: Tag, Not Rewrite](https://mlsanity.com/paper/2005.12592/) (2020)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [Parallel Iterative Edit Models for Local Sequence Transduction](https://mlsanity.com/paper/1910.02893/) (2019)
- [TWEETQA: A Social Media Focused Question Answering Dataset](https://mlsanity.com/paper/1907.06292/) (2019)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [DaN+: Danish Nested Named Entities and Lexical Normalization](https://mlsanity.com/paper/2105.11301/) (2021)
- [Adapting Sequence to Sequence models for Text Normalization in Social Media](https://mlsanity.com/paper/1904.06100/) (2019)
- [Lexical Normalization for Code-switched Data and its Effect on POS-tagging](https://mlsanity.com/paper/2006.01175/) (2020)
- [Sequence-to-Sequence Lexical Normalization with Multilingual Transformers](https://mlsanity.com/paper/2110.02869/) (2021)

## Built on by (most cited first, 7 of 7)

- [Correcting diacritics and typos with a ByT5 transformer model](https://mlsanity.com/paper/2201.13242/) (2022)
- [Neural Text Normalization for Luxembourgish using Real-Life Variation Data](https://mlsanity.com/paper/2412.09383/) (2024)
- [Making Sentence Embeddings Robust to User-Generated Content](https://mlsanity.com/paper/2403.17220/) (2024)
- [Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages](https://mlsanity.com/paper/2505.22273/) (2025)
- [MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages](https://mlsanity.com/paper/2601.16623/) (2026)
- [Identifying and Aligning Medical Claims Made on Social Media with Medical Evidence](https://mlsanity.com/paper/2405.11219/) (2024)
- [The Curse of Multilinguality in Lexical Normalization](https://mlsanity.com/paper/2609.00329/) (2026)

## Abstract (arXiv)

We present the winning entry to the Multilingual Lexical Normalization (MultiLexNorm) shared task at W-NUT 2021 (van der Goot et al., 2021a), which evaluates lexical-normalization systems on 12 social media datasets in 11 languages. We base our solution on a pre-trained byte-level language model, ByT5 (Xue et al., 2021a), which we further pre-train on synthetic data and then fine-tune on authentic normalization data. Our system achieves the best performance by a wide margin in intrinsic evaluation, and also the best performance in extrinsic evaluation through dependency parsing. The source code is released at https://github.com/ufal/multilexnorm2021 and the fine-tuned models at https://huggingface.co/ufal.