mlsanityData to 14 Oct 2026

LangMAP: A Language-Adaptive Approach to Tokenization

arXiv 2606.23566 ↗First posted 22 Jun 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language-specific tokenizers improve tokenization quality and the downstream performance of models on those languages. However, using such a tokenizer comes at a cost: either a new model must be trained from scratch, or the vocabulary of an existing pretrained model must be adapted. We propose Language-adaptive Maximum a Posteriori (LangMAP) Tokenization, a tokenization scheme that extends the UnigramLM algorithm to the multilingual setting, producing language-specific tokenization from a single shared vocabulary. Notably, LangMAP can be used when training a multilingual language model from scratch or to adapt a pretrained model's tokenizer to individual languages without changing its vocabulary. While language labels are required at training time, a key feature of the algorithm is that it then performs language-specific tokenization at inference without knowledge of the input's language. Across 14 open-source tokenizers, 9 natural languages, and 9 programming languages, LangMAP improves morphological boundary alignment and, for all coding languages tested, alignment with abstract syntax tree (AST) leaf boundaries. In fine-tuning experiments, results are mixed: LangMAP improves target-language grammatical acceptability (MultiBLiMP) on the languages tested; its benefits are less consistent on knowledge-related tasks (Global-PIQA, Belebele).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 41 references that are in the map. First posted on arXiv 22 Jun 2026.

Builds on

32 of 41 references.

References · 32

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2023StarCoder: may the source be with you!cited by 799
  5. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  6. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  7. 2024StarCoder 2 and The Stack v2: The Next Generationcited by 502
  8. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
  9. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  10. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  11. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
  12. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
20 more
  1. 2024Tokenization Is More Than Compressioncited by 89
  2. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  3. 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 71
  4. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  5. 2023Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languagescited by 62
  6. 2024Why do language models perform worse for morphologically complex languages?cited by 62
  7. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 36
  8. 2024MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenizationcited by 31
  9. 2021You should evaluate your language model on marginal likelihood over tokenisationscited by 30
  10. 2025Evaluating Morphological Alignment of Tokenizers in 70 Languagescited by 22
  11. 2025Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenizationcited by 20
  12. 2023Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languagescited by 16
  13. 2024Retrofitting Large Language Models with Dynamic Tokenizationcited by 16
  14. 2025Causal Estimation of Tokenisation Biascited by 16
  15. 2026The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?cited by 11
  16. 2024Tokenisation is NP-Completecited by 10
  17. 2025Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignmentcited by 9
  18. 2025Tokenisation over Bounded Alphabets is Hardcited by 5
  19. 2025Which Pieces Does Unigram Tokenization Really Need?cited by 4
  20. 2026What Language is This? Ask Your Tokenizercited by 2

Built on by

Not cited yet.