mlsanityData to 14 Oct 2026

MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling

arXiv 2403.10691 ↗First posted 15 Mar 2024cs.CL cs.AI cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

A major consideration in multilingual language modeling is how to best represent languages with diverse vocabularies and scripts. Although contemporary text encoding methods cover most of the world's writing systems, they exhibit bias towards the high-resource languages of the Global West. As a result, texts of underrepresented languages tend to be segmented into long sequences of linguistically meaningless units. To address the disparities, we introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. Our encoding convention (MYTE) is based on morphemes, as their inventories are more balanced across languages than characters, which are used in previous methods. We show that MYTE produces shorter encodings for all 99 analyzed languages, with the most notable improvements for non-European languages and non-Latin scripts. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). Builds on 31 of its 43 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

31 of 43 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  9. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  10. 2017Word Translation Without Parallel Datacited by 487
  11. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
19 more
  1. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 170
  2. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  3. 2023UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretrainingcited by 151
  4. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  5. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  6. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  7. 2022Efficient Transformers with Dynamic Token Poolingcited by 76
  8. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62
  9. 2018Are All Languages Equally Hard to Language-Model?cited by 53
  10. 2020Improving Multilingual Models with Language-Clustered Vocabulariescited by 53
  11. 2020Morphology Matters: A Multilingual Language Modeling Analysiscited by 37
  12. 2023A Formal Perspective on Byte-Pair Encodingcited by 36
  13. 2023XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languagescited by 34
  14. 2022A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bankcited by 29
  15. 2024A Bit of a Problem: Measurement Disparities in Dataset Sizes Across Languagescited by 25
  16. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  17. 2018Morphological and Language-Agnostic Word Segmentation for NMTcited by 16
  18. 2022A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Modelscited by 12
  19. 2023Effects of sub-word segmentation on performance of transformer language modelscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 14; 2026: 16.

40 papers.

Sort:

By area:

  1. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  2. 2024SpaceByte: Towards Deleting Tokenization from Large Language Modelingcited by 34
  3. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  4. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 28
  5. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  6. 2025Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizationscited by 17
  7. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16
  8. 2024Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensemblescited by 15
  9. 2025Confounding Factors in Relating Model Performance to Morphologycited by 14
  10. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  11. 2025BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenizationcited by 12
  12. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
28 more
  1. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  2. 2024Natural Language Processing RELIES on Linguisticscited by 9
  3. 2025Beyond Literal Token Overlap: Token Alignability for Multilingualitycited by 9
  4. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  5. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  6. 2024Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizerscited by 7
  7. 2025Explaining and Mitigating Crosslingual Tokenizer Inequitiescited by 7
  8. 2025A Partition Cover Approach to Tokenizationcited by 5
  9. 2026Compute Optimal Tokenizationcited by 4
  10. 2025UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8cited by 3
  11. 2026Proxy Compression for Language Modelingcited by 3
  12. 2026Apples to Apples? Towards Comparable Crosslingual Language Model Evaluationcited by 3
  13. 2026Form and Meaning in Intrinsic Multilingual Evaluationscited by 2
  14. 2026Defragmenting Language Models: An Interpretability-based Approach for Vocabulary Expansioncited by 2
  15. 2026Tokenization with Split Treescited by 2
  16. 2026Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabetcited by 2
  17. 2025ByteSpan: Information-Driven Subword Tokenisationcited by 1
  18. 2025Linguistically Informed Tokenization Improves ASR for Underresourced Languagescited by 1
  19. 2026Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Modelscited by 1
  20. 2026Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Modelscited by 1
  21. 2024MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translationcited by 0
  22. 2025From Smor-re-brod to Subwords: Training LLMs on Danish, One Morpheme at a Timecited by 0
  23. 2026Beyond Perplexity: UTF-8 Validity in Byte-aware Language Modelscited by 0
  24. 2026Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Modelscited by 0
  25. 2026Emergent retokenization symmetry in large language models: phenomenology and applicationscited by 0
  26. 2026Dynamic Multi-Byte Prediction With Hierarchical Language Modelscited by 0
  27. 2026What Tokens are Learned when Tokenization is Optimized Jointly with Language Modeling?cited by 0
  28. 2026To Each Language Its Tokenizer: Modular Tokenizers for Efficient Multilingual LLMscited by 0