# Tik-to-Tok: Translating Language Models One Token at a Time: An Embedding Initialization Strategy for Efficient Language Adaptation

arXiv: https://arxiv.org/abs/2310.03477 · HTML: https://mlsanity.com/paper/2310.03477/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 5 Oct 2023.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [CamemBERT: a Tasty French Language Model](https://mlsanity.com/paper/1911.03894/) (2019)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [BPE-Dropout: Simple and Effective Subword Regularization](https://mlsanity.com/paper/1910.13267/) (2019)
- [German's Next Language Model](https://mlsanity.com/paper/2010.10906/) (2020)
- [BERTje: A Dutch BERT Model](https://mlsanity.com/paper/1912.09582/) (2019)
- [Translation Artifacts in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2004.04721/) (2020)
- [RobBERT: a Dutch RoBERTa-based Language Model](https://mlsanity.com/paper/2001.06286/) (2020)
- [As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages](https://mlsanity.com/paper/2012.05628/) (2020)
- [Revisiting Machine Translation for Cross-lingual Classification](https://mlsanity.com/paper/2305.14240/) (2023)
- [Towards Continual Learning for Multilingual Machine Translation via Vocabulary Substitution](https://mlsanity.com/paper/2103.06799/) (2021)
- [SICKNL: A Dataset for Dutch Natural Language Inference](https://mlsanity.com/paper/2101.05716/) (2021)
- [The merits of Universal Language Model Fine-tuning for Small Datasets -- a case with Dutch book reviews](https://mlsanity.com/paper/1910.00896/) (2019)
- [Cross-lingual Transfer of Monolingual Models](https://mlsanity.com/paper/2109.07348/) (2021)
- [Adapting Monolingual Models: Data can be Scarce when Language Similarity is High](https://mlsanity.com/paper/2105.02855/) (2021)
- [One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages](https://mlsanity.com/paper/2102.02585/) (2021)

## Built on by (most cited first, 11 of 11)

- [Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP](https://mlsanity.com/paper/2408.04303/) (2024)
- [Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?](https://mlsanity.com/paper/2410.09644/) (2024)
- [The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems](https://mlsanity.com/paper/2602.15382/) (2026)
- [Model-Aware Tokenizer Transfer](https://mlsanity.com/paper/2510.21954/) (2025)
- [Bilingual BSARD: Extending Statutory Article Retrieval to Dutch](https://mlsanity.com/paper/2412.07462/) (2024)
- [ChocoLlama: Lessons Learned From Teaching Llamas Dutch](https://mlsanity.com/paper/2412.07633/) (2024)
- [MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch](https://mlsanity.com/paper/2509.12340/) (2025)
- [Writing-System-Level Tokenizer Adaptation for Byte-Level BPE](https://mlsanity.com/paper/2608.00582/) (2026)
- [Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages](https://mlsanity.com/paper/2506.19187/) (2025)
- [Conditional Unigram Tokenization with Parallel Data](https://mlsanity.com/paper/2507.07824/) (2025)
- [KazByte: Adapting Qwen models to Kazakh via Byte-level Adapter](https://mlsanity.com/paper/2603.27859/) (2026)

## Abstract (arXiv)

Training monolingual language models for low and mid-resource languages is made challenging by limited and often inadequate pretraining data. In this study, we propose a novel model conversion strategy to address this issue, adapting high-resources monolingual language models to a new target language. By generalizing over a word translation dictionary encompassing both the source and target languages, we map tokens from the target tokenizer to semantically similar tokens from the source language tokenizer. This one-to-many token mapping improves tremendously the initialization of the embedding table for the target language. We conduct experiments to convert high-resource models to mid- and low-resource languages, namely Dutch and Frisian. These converted models achieve a new state-of-the-art performance on these languages across all sorts of downstream tasks. By reducing significantly the amount of data and time required for training state-of-the-art models, our novel model conversion strategy has the potential to benefit many languages worldwide.