mlsanityData to 14 Oct 2026

Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5

arXiv 2410.11627 ↗First posted 15 Oct 2024cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Morphology is a crucial factor for multilingual language modeling as it poses direct challenges for tokenization. Here, we seek to understand how tokenization influences the morphological knowledge encoded in multilingual language models. Specifically, we capture the impact of tokenization by contrasting two multilingual language models: mT5 and ByT5. The two models share the same architecture, training objective, and training data and only differ in their tokenization strategies: subword tokenization vs.@ character-level tokenization. Probing the morphological knowledge encoded in these models on four tasks and 17 languages, our analyses show that the models learn the morphological systems of some languages better than others and that morphological information is encoded in the middle and late layers. Finally, we show that languages with more irregularities benefit more from having a higher share of the pre-training data.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 50 of its 79 references that are in the map. First posted on arXiv 15 Oct 2024.

Builds on

50 of 79 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  9. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  12. 2019Cross-lingual Language Model Pretrainingcited by 1,090
38 more
  1. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  2. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  3. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  4. 2019Designing and Interpreting Probes with Control Taskscited by 640
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  7. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  8. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  9. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  10. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  11. 2020Are All Languages Created Equal in Multilingual BERT?cited by 207
  12. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
  13. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  14. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  15. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 143
  16. 2015Character-Aware Neural Language Modelscited by 132
  17. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 120
  18. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  19. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 98
  20. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  21. 2021Conditional probing: measuring usable information beyond a baselinecited by 62
  22. 2018Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Taskscited by 56
  23. 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
  24. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  25. 2019What Kind of Language Is Hard to Language-Model?cited by 50
  26. 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
  27. 2019On the Linguistic Representational Power of Neural Machine Translation Modelscited by 34
  28. 2018Assessing Composition in Sentence Vector Representationscited by 26
  29. 2016A Character-Level Decoder without Explicit Segmentation for Neural Machine Translationcited by 21
  30. 2020A Systematic Analysis of Morphological Content in BERT Models for Multiple Languagescited by 20
  31. 2021Subword Pooling Makes a Differencecited by 18
  32. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 15
  33. 2019Morphological Irregularity Correlates with Frequencycited by 13
  34. 2023What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structurecited by 13
  35. 2020Character-Level Translation with Self-attentioncited by 12
  36. 2016Word Representation Models for Morphologically Rich Languages in Neural Machine Translationcited by 9
  37. 2023Morphosyntactic probing of multilingual BERT modelscited by 8
  38. 2023Toucan: Token-Aware Character Level Language Modelingcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 6.

14 papers.

Sort:

By area:

  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2025Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?cited by 9
  3. 2025Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Modelscited by 5
  4. 2025MUTANT: A Recipe for Multilingual Tokenizer Designcited by 4
  5. 2025TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behaviorcited by 4
  6. 2026The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Modelscited by 4
  7. 2025UNVEILING: What Makes Linguistics Olympiad Puzzles Tricky for LLMs?cited by 3
  8. 2025Community size rather than grammatical complexity better predicts Large Language Model accuracy in a novel Wug Testcited by 2
  9. 2026Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrewcited by 2
  10. 2025Orthographic Constraint Satisfaction and Human Difficulty Alignment in Large Language Modelscited by 0
  11. 2025FastPOS: Language-Agnostic Scalable POS Tagging Framework Low-Resource Use Casecited by 0
  12. 2026Lost in the Tower of Babel: The Adverse Effects of Incidental Multilingualism in LLMscited by 0
2 more
  1. 2026When Tokenization is Secretly Output Supervisioncited by 0
  2. 2026The Limits of BPE Tokenization in Polish: Segmentation-Flexional Forms, Grammatical Anchoring, and First-Person Stability in Inflectional Language Modelscited by 0