# Don't Touch My Diacritics

arXiv: https://arxiv.org/abs/2410.24140 · HTML: https://mlsanity.com/paper/2410.24140/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 31 Oct 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains](https://mlsanity.com/paper/2402.10373/) (2024)
- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [How to Train BERT with an Academic Budget](https://mlsanity.com/paper/2104.07705/) (2021)
- [GREEK-BERT: The Greeks visiting Sesame Street](https://mlsanity.com/paper/2008.12014/) (2020)
- [Text Rendering Strategies for Pixel Language Models](https://mlsanity.com/paper/2311.00522/) (2023)
- [Restoring Hebrew Diacritics Without a Dictionary](https://mlsanity.com/paper/2105.05209/) (2021)
- [Analyzing Gender Representation in Multilingual Models](https://mlsanity.com/paper/2204.09168/) (2022)

## Built on by (most cited first, 7 of 7)

- [Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier](https://mlsanity.com/paper/2504.00178/) (2025)
- [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026)
- [Hebrew Diacritics Restoration using Visual Representation](https://mlsanity.com/paper/2510.26521/) (2025)
- [Diacritic Restoration for Low-Resource Indigenous Languages: Case Study with Bribri and Cook Islands M=aori](https://mlsanity.com/paper/2512.19630/) (2025)
- [The Degree of Language Diacriticity and Its Effect on Tasks](https://mlsanity.com/paper/2603.27653/) (2026)
- [textitVersteasch du mi? Computational and Socio-Linguistic Perspectives on GenAI, LLMs, and Non-Standard Language](https://mlsanity.com/paper/2603.28213/) (2026)
- [The Functionalizer: Lossless Functional Decomposition for Subword Tokenization](https://mlsanity.com/paper/2609.15991/) (2026)

## Abstract (arXiv)

The common practice of preprocessing text before feeding it into NLP models introduces many decision points which have unintended consequences on model performance. In this opinion piece, we focus on the handling of diacritics in texts originating in many languages and scripts. We demonstrate, through several case studies, the adverse effects of inconsistent encoding of diacritized characters and of removing diacritics altogether. We call on the community to adopt simple but necessary steps across all models and toolkits in order to improve handling of diacritized text and, by extension, increase equity in multilingual NLP.