mlsanityData to 14 Oct 2026

Model-Aware Tokenizer Transfer

arXiv 2510.21954 ↗First posted 24 Oct 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are trained to support an increasing number of languages, yet their predefined tokenizers remain a bottleneck for adapting models to lower-resource or distinct-script languages. Existing tokenizer transfer methods typically rely on semantic heuristics to initialize new embeddings, ignoring higher-layer model dynamics and limiting transfer quality. We propose Model-Aware Tokenizer Transfer (MATT), a method that incorporates model internals into the tokenizer transfer process. MATT introduces an Attention Influence Modeling (AIM) objective that distills inter-token communication patterns from a source model into a target model with a new tokenizer, providing an efficient warm-up before standard language modeling. Unlike approaches that focus solely on embedding similarity, MATT leverages attention behavior to guide embedding initialization and adaptation. Experiments across diverse linguistic settings show that MATT recovers a large fraction of the original model's performance within a few GPU hours, outperforming heuristic baselines. These results demonstrate that incorporating model-level signals offers a practical and effective path toward robust tokenizer transfer in multilingual LLMs.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

47 of 59 references.

References · 47

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2023Qwen Technical Reportcited by 3,344
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2025Gemma 3 Technical Reportcited by 1,824
  8. 2016Enriching Word Vectors with Subword Informationcited by 1,583
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  10. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  11. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  12. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
35 more
  1. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  2. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  4. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  5. 2023The Falcon Series of Open Language Modelscited by 535
  6. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  7. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  8. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  9. 2024Byte Latent Transformer: Patches Scale Better Than Tokenscited by 146
  10. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 129
  11. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 108
  12. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 92
  13. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  14. 2023XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Modelscited by 88
  15. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 72
  16. 2019The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-Englishcited by 62
  17. 2020As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languagescited by 54
  18. 2024Understanding Factual Recall in Transformers via Associative Memoriescited by 51
  19. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
  20. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  21. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 36
  22. 2024Zero-Shot Tokenizer Transfercited by 31
  23. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26
  24. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  25. 2025An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)cited by 16
  26. 2024Bielik 7B v0.1: A Polish Language Model -- Development, Insights, and Evaluationcited by 15
  27. 2021Cross-lingual Transfer of Monolingual Modelscited by 14
  28. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  29. 2023MAUPQA: Massive Automatically-created Polish Question Answering Datasetcited by 11
  30. 2023Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptationcited by 11
  31. 2024Large Vocabulary Size Improves Large Language Modelscited by 11
  32. 2024Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languagescited by 11
  33. 2025TokAlign: Efficient Vocabulary Adaptation via Token Alignmentcited by 11
  34. 2025Token Distillation: Attention-aware Input Embeddings For New Tokenscited by 6
  35. 2025HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initializationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  2. 2026Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Seriescited by 1
  3. 2026Writing-System-Level Tokenizer Adaptation for Byte-Level BPEcited by 1
  4. 2026KazByte: Adapting Qwen models to Kazakh via Byte-level Adaptercited by 0
  5. 2026Disentangling Language Modeling and Boundariescited by 0
  6. 2026Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relaycited by 0