# Diacritics Restoration using BERT with Analysis on Czech language

arXiv: https://arxiv.org/abs/2105.11408 · HTML: https://mlsanity.com/paper/2105.11408/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 24 May 2021.

## Builds on (most cited first, 8 of 8)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection](https://mlsanity.com/paper/2004.10643/) (2020)
- [Attentive Sequence-to-Sequence Learning for Diacritic Restoration of Yor`ub'a Language Text](https://mlsanity.com/paper/1804.00832/) (2018)
- [Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization](https://mlsanity.com/paper/2011.00538/) (2020)
- [Efficient Convolutional Neural Networks for Diacritic Restoration](https://mlsanity.com/paper/1912.06900/) (2019)
- [Czech Text Processing with Contextual Embeddings: POS Tagging, Lemmatization, Parsing and NER](https://mlsanity.com/paper/1909.03544/) (2019)

## Built on by (most cited first, 6 of 6)

- [Improving Arabic Diacritization by Learning to Diacritize and Translate](https://mlsanity.com/paper/2109.14150/) (2021)
- [Correcting diacritics and typos with a ByT5 transformer model](https://mlsanity.com/paper/2201.13242/) (2022)
- [YAD: Leveraging T5 for Improved Automatic Diacritization of Yor`ub'a Text](https://mlsanity.com/paper/2412.20218/) (2024)
- [Dilated Convolutional Neural Networks for Lightweight Diacritics Restoration](https://mlsanity.com/paper/2201.06757/) (2022)
- [Hebrew Diacritics Restoration using Visual Representation](https://mlsanity.com/paper/2510.26521/) (2025)
- [The Degree of Language Diacriticity and Its Effect on Tasks](https://mlsanity.com/paper/2603.27653/) (2026)

## Abstract (arXiv)

We propose a new architecture for diacritics restoration based on contextualized embeddings, namely BERT, and we evaluate it on 12 languages with diacritics. Furthermore, we conduct a detailed error analysis on Czech, a morphologically rich language with a high level of diacritization. Notably, we manually annotate all mispredictions, showing that roughly 44% of them are actually not errors, but either plausible variants (19%), or the system corrections of erroneous data (25%). Finally, we categorize the real errors in detail. We release the code at https://github.com/ufal/bert-diacritics-restoration.