# CATT: Character-based Arabic Tashkeel Transformer

arXiv: https://arxiv.org/abs/2407.03236 · HTML: https://mlsanity.com/paper/2407.03236/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 3 Jul 2024.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Fully Character-Level Neural Machine Translation without Explicit Segmentation](https://mlsanity.com/paper/1610.03017/) (2016)
- [Arabic Text Diacritization Using Deep Neural Networks](https://mlsanity.com/paper/1905.01965/) (2019)
- [Neural Arabic Text Diacritization: State of the Art Results and a Novel Approach for Machine Translation](https://mlsanity.com/paper/1911.03531/) (2019)
- [Deep Diacritization: Efficient Hierarchical Recurrence for Improved Arabic Diacritization](https://mlsanity.com/paper/2011.00538/) (2020)
- [Arabic Diacritic Recovery Using a Feature-Rich biLSTM Model](https://mlsanity.com/paper/2002.01207/) (2020)
- [Block-Normalized Gradient Method: An Empirical Study for Training Deep Neural Network](https://mlsanity.com/paper/1707.04822/) (2017)
- [Arabic Text Diacritization In The Age Of Transfer Learning: Token Classification Is All You Need](https://mlsanity.com/paper/2401.04848/) (2024)
- [Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization](https://mlsanity.com/paper/2303.14588/) (2023)

## Built on by (most cited first, 4 of 4)

- [NADI 2025: The First Multidialectal Arabic Speech Processing Shared Task](https://mlsanity.com/paper/2509.02038/) (2025)
- [Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization](https://mlsanity.com/paper/2605.25928/) (2026)
- [CANDLE: CTC-based Arabic Noisy-character Deduplication using a Lightweight Encoder](https://mlsanity.com/paper/2606.24758/) (2026)
- [SHAMS: An Audio-Grounded Pronunciation Benchmark for Levantine Arabic](https://mlsanity.com/paper/2610.01427/) (2026)

## Abstract (arXiv)

Tashkeel, or Arabic Text Diacritization (ATD), greatly enhances the comprehension of Arabic text by removing ambiguity and minimizing the risk of misinterpretations caused by its absence. It plays a crucial role in improving Arabic text processing, particularly in applications such as text-to-speech and machine translation. This paper introduces a new approach to training ATD models. First, we finetuned two transformers, encoder-only and encoder-decoder, that were initialized from a pretrained character-based BERT. Then, we applied the Noisy-Student approach to boost the performance of the best model. We evaluated our models alongside 11 commercial and open-source models using two manually labeled benchmark datasets: WikiNews and our CATT dataset. Our findings show that our top model surpasses all evaluated models by relative Diacritic Error Rates (DERs) of 30.83% and 35.21% on WikiNews and CATT, respectively, achieving state-of-the-art in ATD. In addition, we show that our model outperforms GPT-4-turbo on CATT dataset by a relative DER of 9.36%. We open-source our CATT models and benchmark dataset for the research communityfootnotehttps://github.com/abjadai/catt.