# Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models arXiv: https://arxiv.org/abs/2403.11130 ยท HTML: https://mlsanity.com/paper/2403.11130/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 30 of its 51 references that are in the map. - First posted on arXiv 17 Mar 2024. ## Builds on (most cited first, 30 of 30) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013) - [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015) - [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016) - [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015) - [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018) - [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021) - [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020) - [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://mlsanity.com/paper/2012.15613/) (2020) - [Byte Pair Encoding is Suboptimal for Language Model Pretraining](https://mlsanity.com/paper/2004.03720/) (2020) - [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021) - [Charformer: Fast Character Transformers via Gradient-based Subword Tokenization](https://mlsanity.com/paper/2106.12672/) (2021) - [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022) - [Tokenizer Choice For LLM Training: Negligible or Crucial?](https://mlsanity.com/paper/2310.08754/) (2023) - [KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Media](https://mlsanity.com/paper/2007.13184/) (2020) - [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022) - [FRAGE: Frequency-Agnostic Word Representation](https://mlsanity.com/paper/1809.06858/) (2018) - [Arabic Dialect Identification in the Wild](https://mlsanity.com/paper/2005.06557/) (2020) - [What Makes for Good Visual Tokenizers for Large Language Models?](https://mlsanity.com/paper/2305.12223/) (2023) - [From English To Foreign Languages: Transferring Pre-trained Language Models](https://mlsanity.com/paper/2002.07306/) (2020) - [A Survey on Transfer Learning in Natural Language Processing](https://mlsanity.com/paper/2007.04239/) (2020) - [An Empirical Study of Tokenization Strategies for Various Korean NLP Tasks](https://mlsanity.com/paper/2010.02534/) (2020) - [A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translation](https://mlsanity.com/paper/1905.10453/) (2019) - [WangchanBERTa: Pretraining transformer-based Thai Language Models](https://mlsanity.com/paper/2101.09635/) (2021) - [Bridging the Gap for Tokenizer-Free Language Models](https://mlsanity.com/paper/1908.10322/) (2019) - [Joint Optimization of Tokenization and Downstream Model](https://mlsanity.com/paper/2105.12410/) (2021) - [Evaluating Various Tokenizers for Arabic Text Classification](https://mlsanity.com/paper/2106.07540/) (2021) - [The Impact of Preprocessing on Arabic-English Statistical and Neural Machine Translation](https://mlsanity.com/paper/1906.11751/) (2019) ## Built on by (most cited first, 5 of 5) - [The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?](https://mlsanity.com/paper/2601.07220/) (2026) - [How does a Language-Specific Tokenizer affect LLMs?](https://mlsanity.com/paper/2502.12560/) (2025) - [Can Perplexity Predict Fine-tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali](https://mlsanity.com/paper/2404.18071/) (2024) - [Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language](https://mlsanity.com/paper/2410.03718/) (2024) - [Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic](https://mlsanity.com/paper/2510.17001/) (2025) ## Abstract (arXiv) This paper presents a comprehensive examination of the impact of tokenization strategies and vocabulary sizes on the performance of Arabic language models in downstream natural language processing tasks. Our investigation focused on the effectiveness of four tokenizers across various tasks, including News Classification, Hate Speech Detection, Sentiment Analysis, and Natural Language Inference. Leveraging a diverse set of vocabulary sizes, we scrutinize the intricate interplay between tokenization approaches and model performance. The results reveal that Byte Pair Encoding (BPE) with Farasa outperforms other strategies in multiple tasks, underscoring the significance of morphological analysis in capturing the nuances of the Arabic language. However, challenges arise in sentiment analysis, where dialect specific segmentation issues impact model efficiency. Computational efficiency analysis demonstrates the stability of BPE with Farasa, suggesting its practical viability. Our study uncovers limited impacts of vocabulary size on model performance while keeping the model size unchanged. This is challenging the established beliefs about the relationship between vocabulary, model size, and downstream tasks, emphasizing the need for the study of models' size and their corresponding vocabulary size to generalize across domains and mitigate biases, particularly in dialect based datasets. Paper's recommendations include refining tokenization strategies to address dialect challenges, enhancing model robustness across diverse linguistic contexts, and expanding datasets to encompass the rich dialect based Arabic. This work not only advances our understanding of Arabic language models but also lays the foundation for responsible and ethical developments in natural language processing technologies tailored to the intricacies of the Arabic language.