# Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization

arXiv: https://arxiv.org/abs/2303.14588 · HTML: https://mlsanity.com/paper/2303.14588/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 25 Mar 2023.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Scaling Laws for Transfer](https://mlsanity.com/paper/2102.01293/) (2021)
- [Semi-supervised sequence tagging with bidirectional language models](https://mlsanity.com/paper/1705.00108/) (2017)
- [AraT5: Text-to-Text Transformers for Arabic Language Generation](https://mlsanity.com/paper/2109.12068/) (2021)
- [Bridging the Gap for Tokenizer-Free Language Models](https://mlsanity.com/paper/1908.10322/) (2019)
- [Arabic Text Diacritization Using Deep Neural Networks](https://mlsanity.com/paper/1905.01965/) (2019)
- [Correcting diacritics and typos with a ByT5 transformer model](https://mlsanity.com/paper/2201.13242/) (2022)

## Built on by (most cited first, 1 of 1)

- [CATT: Character-based Arabic Tashkeel Transformer](https://mlsanity.com/paper/2407.03236/) (2024)

## Abstract (arXiv)

Most of previous work on learning diacritization of the Arabic language relied on training models from scratch. In this paper, we investigate how to leverage pre-trained language models to learn diacritization. We finetune token-free pre-trained multilingual models (ByT5) to learn to predict and insert missing diacritics in Arabic text, a complex task that requires understanding the sentence semantics and the morphological structure of the tokens. We show that we can achieve state-of-the-art on the diacritization task with minimal amount of training and no feature engineering, reducing WER by 40%. We release our finetuned models for the greater benefit of the researchers in the community.