# Correcting diacritics and typos with a ByT5 transformer model

arXiv: https://arxiv.org/abs/2201.13242 · HTML: https://mlsanity.com/paper/2201.13242/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 118 references that are in the map.
- First posted on arXiv 31 Jan 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [A Simple Recipe for Multilingual Grammatical Error Correction](https://mlsanity.com/paper/2106.03830/) (2021)
- [GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors](https://mlsanity.com/paper/1911.12893/) (2019)
- [A Multitask Learning Approach for Diacritic Restoration](https://mlsanity.com/paper/2006.04016/) (2020)
- [Attentive Sequence-to-Sequence Learning for Diacritic Restoration of Yor`ub'a Language Text](https://mlsanity.com/paper/1804.00832/) (2018)
- [Restoring Hebrew Diacritics Without a Dictionary](https://mlsanity.com/paper/2105.05209/) (2021)
- [Efficient Convolutional Neural Networks for Diacritic Restoration](https://mlsanity.com/paper/1912.06900/) (2019)
- [Spelling Correction with Denoising Transformer](https://mlsanity.com/paper/2105.05977/) (2021)
- ['UFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5](https://mlsanity.com/paper/2110.15248/) (2021)
- [Diacritics Restoration using BERT with Analysis on Czech language](https://mlsanity.com/paper/2105.11408/) (2021)
- [Correcting the Autocorrect: Context-Aware Typographical Error Correction via Training Data Augmentation](https://mlsanity.com/paper/2005.01158/) (2020)
- [Hierarchical Transformer Encoders for Vietnamese Spelling Correction](https://mlsanity.com/paper/2105.13578/) (2021)
- [Romanian Diacritics Restoration Using Recurrent Neural Networks](https://mlsanity.com/paper/2009.02743/) (2020)

## Built on by (most cited first, 6 of 6)

- [Byte-Level Grammatical Error Correction Using Synthetic and Curated Corpora](https://mlsanity.com/paper/2305.17906/) (2023)
- [YAD: Leveraging T5 for Improved Automatic Diacritization of Yor`ub'a Text](https://mlsanity.com/paper/2412.20218/) (2024)
- [Towards Lithuanian grammatical error correction](https://mlsanity.com/paper/2203.09963/) (2022)
- [Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization](https://mlsanity.com/paper/2303.14588/) (2023)
- [Domain specificity and data efficiency in typo tolerant spell checkers: the case of search in online marketplaces](https://mlsanity.com/paper/2308.01976/) (2023)
- [TDR: Tagalog Diacritic Restoration Dataset](https://mlsanity.com/paper/2610.17597/) (2026)

## Abstract (arXiv)

Due to the fast pace of life and online communications and the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing in other languages. Restoring diacritics and correcting spelling is important for proper language use and the disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately: the state-of-the-art diacritics restoration methods do not tolerate other typos, but classical spellcheckers also cannot deal adequately with all the diacritics missing. In this work, we tackle both problems at once by employing the newly-developed universal ByT5 byte-level seq2seq transformer model that requires no language-specific model structures. For a comparison, we perform diacritics restoration on benchmark datasets of 12 languages, with the addition of Lithuanian. The experimental investigation proves that our approach is able to achieve results (> 98%) comparable to the previous state-of-the-art, despite being trained less and on fewer data. Our approach is also able to restore diacritics in words not seen during training with > 76% accuracy. Our simultaneous diacritics restoration and typos correction approach reaches > 94% alpha-word accuracy on the 13 languages. It has no direct competitors and strongly outperforms classical spell-checking or dictionary-based approaches. We also demonstrate all the accuracies to further improve with more training. Taken together, this shows the great real-world application potential of our suggested methods to more data, languages, and error classes.