mlsanityData to 14 Oct 2026

Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models

arXiv 2403.11130 ↗First posted 17 Mar 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper presents a comprehensive examination of the impact of tokenization strategies and vocabulary sizes on the performance of Arabic language models in downstream natural language processing tasks. Our investigation focused on the effectiveness of four tokenizers across various tasks, including News Classification, Hate Speech Detection, Sentiment Analysis, and Natural Language Inference. Leveraging a diverse set of vocabulary sizes, we scrutinize the intricate interplay between tokenization approaches and model performance. The results reveal that Byte Pair Encoding (BPE) with Farasa outperforms other strategies in multiple tasks, underscoring the significance of morphological analysis in capturing the nuances of the Arabic language. However, challenges arise in sentiment analysis, where dialect specific segmentation issues impact model efficiency. Computational efficiency analysis demonstrates the stability of BPE with Farasa, suggesting its practical viability. Our study uncovers limited impacts of vocabulary size on model performance while keeping the model size unchanged. This is challenging the established beliefs about the relationship between vocabulary, model size, and downstream tasks, emphasizing the need for the study of models' size and their corresponding vocabulary size to generalize across domains and mitigate biases, particularly in dialect based datasets. Paper's recommendations include refining tokenization strategies to address dialect challenges, enhancing model robustness across diverse linguistic contexts, and expanding datasets to encompass the rich dialect based Arabic. This work not only advances our understanding of Arabic language models but also lays the foundation for responsible and ethical developments in natural language processing technologies tailored to the intricacies of the Arabic language.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 30 of its 51 references that are in the map. First posted on arXiv 17 Mar 2024.

Builds on

30 of 51 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  7. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  8. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  10. 2020AraBERT: Transformer-based Model for Arabic Language Understandingcited by 362
  11. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  12. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
18 more
  1. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  2. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  3. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  4. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  5. 2020KUISAIL at SemEval-2020 Task 12: BERT-CNN for Offensive Speech Identification in Social Mediacited by 84
  6. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  7. 2018FRAGE: Frequency-Agnostic Word Representationcited by 47
  8. 2020Arabic Dialect Identification in the Wildcited by 45
  9. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  10. 2020From English To Foreign Languages: Transferring Pre-trained Language Modelscited by 37
  11. 2020A Survey on Transfer Learning in Natural Language Processingcited by 33
  12. 2020An Empirical Study of Tokenization Strategies for Various Korean NLP Taskscited by 33
  13. 2019A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translationcited by 30
  14. 2021WangchanBERTa: Pretraining transformer-based Thai Language Modelscited by 26
  15. 2019Bridging the Gap for Tokenizer-Free Language Modelscited by 21
  16. 2021Joint Optimization of Tokenization and Downstream Modelcited by 15
  17. 2021Evaluating Various Tokenizers for Arabic Text Classificationcited by 14
  18. 2019The Impact of Preprocessing on Arabic-English Statistical and Neural Machine Translationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2025How does a Language-Specific Tokenizer affect LLMs?cited by 6
  3. 2024Can Perplexity Predict Fine-tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepalicited by 3
  4. 2024Performance Evaluation of Tokenizers in Large Language Models for the Assamese Languagecited by 3
  5. 2025Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmeticcited by 1