mlsanityData to 14 Oct 2026

How do different tokenizers perform on downstream tasks in scriptio continua languages?: A case study in Japanese

arXiv 2306.09572 ↗First posted 16 Jun 2023cs.CL cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

This paper investigates the effect of tokenizers on the downstream performance of pretrained language models (PLMs) in scriptio continua languages where no explicit spaces exist between words, using Japanese as a case study. The tokenizer for such languages often consists of a morphological analyzer and a subword tokenizer, requiring us to conduct a comprehensive study of all possible pairs. However, previous studies lack this comprehensiveness. We therefore train extensive sets of tokenizers, build a PLM using each, and measure the downstream performance on a wide range of tasks. Our results demonstrate that each downstream task has a different optimal morphological analyzer, and that it is better to use Byte-Pair-Encoding or Unigram rather than WordPiece as a subword tokenizer, regardless of the type of task.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

20 of 40 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  12. 2016Deep Biaffine Attention for Neural Dependency Parsingcited by 334
8 more
  1. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  2. 2020Byte Pair Encoding is Suboptimal for Language Model Pretrainingcited by 192
  3. 2020The Multilingual Amazon Reviews Corpuscited by 143
  4. 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
  5. 2021Your fairness may vary: Pretrained language model fairness in toxic text classificationcited by 39
  6. 2020An Empirical Study of Tokenization Strategies for Various Korean NLP Taskscited by 33
  7. 2021WangchanBERTa: Pretraining transformer-based Thai Language Modelscited by 26
  8. 2022Improving Tokenisation by Alternative Treatment of Spacescited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 8; 2026: 2.

20 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 254
  2. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  3. 2024How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?cited by 22
  4. 2024The Foundations of Tokenization: Statistical and Computational Concernscited by 22
  5. 2024An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inferencecited by 18
  6. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16
  7. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  8. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  9. 2024Efficient Continual Pre-training of LLMs for Low-resource Languagescited by 8
  10. 2025Tokenization Standards for Linguistic Integrity: Turkish as a Benchmarkcited by 8
  11. 2025How does a Language-Specific Tokenizer affect LLMs?cited by 6
  12. 2025Overcoming Vocabulary Constraints with Pixel-level Fallbackcited by 6
8 more
  1. 2025Dougal Dil .Icslemede Tokenizasyon Standartlari ve "Olcc"um"u: T"urkcce "Uzerinden B"uy"uk Dil Modellerinin Karcsilacstirmali Analizicited by 4
  2. 2024Responsible Multilingual Large Language Models: A Survey of Development, Applications, and Societal Impactcited by 3
  3. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  4. 2025Less Is More: Reducing Token Counts Without Compromising Performancecited by 2
  5. 2024How Important Is Tokenization in French Medical Masked Language Models?cited by 0
  6. 2024Efficacy of ByT5 in Multilingual Translation of Biblical Texts for Underrepresented Languagescited by 0
  7. 2025Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-servicecited by 0
  8. 2026LGSE: Lexically Grounded Subword Embedding Initialization for Low-Resource Language Adaptationcited by 0